{
  "cells": [
    {
      "cell_type": "raw",
      "id": "98e133ed-7f91-4e11-9f6b-abde9a67f980",
      "metadata": {},
      "source": [
        "# Specifying and estimating a linkage model\n",
        "\n",
        "<a target=\"_blank\" href=\"https://colab.research.google.com/github/moj-analytical-services/splink/blob/master/docs/demos/tutorials/04_Estimating_model_parameters.ipynb\">\n",
        "  <img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/>\n",
        "</a>\n",
        "\n",
        "In the last tutorial we looked at how we can use blocking rules to generate pairwise record comparisons.\n",
        "\n",
        "Now it's time to estimate a probabilistic linkage model to score each of these comparisons. The resultant match score is a prediction of whether the two records represent the same entity (e.g. are the same person).\n",
        "\n",
        "The purpose of estimating the model is to learn the relative importance of different parts of your data for the purpose of data linking.\n",
        "\n",
        "For example, a match on date of birth is a much stronger indicator that two records refer to the same entity than a match on gender. A mismatch on gender may be a stronger indicate against two records referring than a mismatch on name, since names are more likely to be entered differently.\n",
        "\n",
        "The relative importance of different information is captured in the (partial) 'match weights', which can be learned from your data. These match weights are then added up to compute the overall match score.\n",
        "\n",
        "The match weights are are derived from the `m` and `u` parameters of the underlying Fellegi Sunter model. Splink uses various statistical routines to estimate these parameters. Further details of the underlying theory can be found [here](https://www.robinlinacre.com/intro_to_probabilistic_linkage/), which will help you understand this part of the tutorial.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "0f104340",
      "metadata": {},
      "source": [
        "## Specifying a linkage model\n",
        "\n",
        "To build a linkage model, the user defines the partial match weights that `splink` needs to estimate. This is done by defining how the information in the input records should be compared.\n",
        "\n",
        "To be concrete, here is an example comparison:\n",
        "\n",
        "| first_name_l | first_name_r | surname_l | surname_r | dob_l      | dob_r      | city_l | city_r | email_l             | email_r             |\n",
        "| ------------ | ------------ | --------- | --------- | ---------- | ---------- | ------ | ------ | ------------------- | ------------------- |\n",
        "| Robert       | Rob          | Allen     | Allen     | 1971-05-24 | 1971-06-24 | nan    | London | roberta25@smith.net | roberta25@smith.net |\n",
        "\n",
        "What functions should we use to assess the similarity of `Rob` vs. `Robert` in the the `first_name` field?\n",
        "\n",
        "Should similarity in the `dob` field be computed in the same way, or a different way?\n",
        "\n",
        "Your job as the developer of a linkage model is to decide what comparisons are most appropriate for the types of data you have.\n",
        "\n",
        "Splink can then estimate how much weight to place on a fuzzy match of `Rob` vs. `Robert`, relative to an exact match on `Robert`, or a non-match.\n",
        "\n",
        "Defining these scenarios is done using `Comparison`s.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "8a520392",
      "metadata": {},
      "source": [
        "### Comparisons\n",
        "\n",
        "The concept of a `Comparison` has a specific definition within Splink: it defines how data from one or more input columns is compared.\n",
        "\n",
        "For example, one `Comparison` may represent how similarity is assessed for a person's date of birth.\n",
        "\n",
        "Another `Comparison` may represent the comparison of a person's name or location.\n",
        "\n",
        "A model is composed of many `Comparison`s, which between them assess the similarity of all of the columns being used for data linking.\n",
        "\n",
        "Each `Comparison` contains two or more `ComparisonLevels` which define _n_ discrete gradations of similarity between the input columns within the Comparison.\n",
        "\n",
        "As such `ComparisonLevels`are nested within `Comparisons` as follows:\n",
        "\n",
        "```\n",
        "Data Linking Model\n",
        "├─-- Comparison: Date of birth\n",
        "│    ├─-- ComparisonLevel: Exact match\n",
        "│    ├─-- ComparisonLevel: One character difference\n",
        "│    ├─-- ComparisonLevel: All other\n",
        "├─-- Comparison: Surname\n",
        "│    ├─-- ComparisonLevel: Exact match on surname\n",
        "│    ├─-- ComparisonLevel: All other\n",
        "│    etc.\n",
        "```\n",
        "\n",
        "Our example data would therefore result in the following comparisons, for `dob` and `surname`:\n",
        "\n",
        "| dob_l      | dob_r      | comparison_level         | interpretation |\n",
        "| ---------- | ---------- | ------------------------ | -------------- |\n",
        "| 1971-05-24 | 1971-05-24 | Exact match              | great match    |\n",
        "| 1971-05-24 | 1971-06-24 | One character difference | fuzzy match    |\n",
        "| 1971-05-24 | 2000-01-02 | All other                | bad match      |\n",
        "\n",
        "<br/>\n",
        "\n",
        "| surname_l | surname_r | comparison_level | interpretation                                        |\n",
        "| --------- | --------- | ---------------- | ----------------------------------------------------- |\n",
        "| Rob       | Rob       | Exact match      | great match                                           |\n",
        "| Rob       | Jane      | All other        | bad match                                             |\n",
        "| Rob       | Robert    | All other        | bad match, this comparison has no notion of nicknames |\n",
        "\n",
        "More information about specifying comparisons can be found [here](../../topic_guides/comparisons/customising_comparisons.ipynb) and [here](../../topic_guides/comparisons/comparisons_and_comparison_levels.md).\n",
        "\n",
        "We will now use these concepts to build a data linking model.\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 1,
      "id": "9ceef6f1",
      "metadata": {
        "execution": {
          "iopub.execute_input": "2024-07-11T13:34:21.934832Z",
          "iopub.status.busy": "2024-07-11T13:34:21.934445Z",
          "iopub.status.idle": "2024-07-11T13:34:21.956453Z",
          "shell.execute_reply": "2024-07-11T13:34:21.955605Z"
        },
        "tags": [
          "hide_input"
        ]
      },
      "outputs": [],
      "source": [
        "# Uncomment and run this cell if you're running in Google Colab.\n",
        "# !pip install splink"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 2,
      "id": "aa6a9e30",
      "metadata": {
        "execution": {
          "iopub.execute_input": "2024-07-11T13:34:21.961236Z",
          "iopub.status.busy": "2024-07-11T13:34:21.960890Z",
          "iopub.status.idle": "2024-07-11T13:34:23.786243Z",
          "shell.execute_reply": "2024-07-11T13:34:23.785015Z"
        },
        "tags": []
      },
      "outputs": [],
      "source": [
        "# Begin by reading in the tutorial data again\n",
        "from splink import splink_datasets\n",
        "\n",
        "df = splink_datasets.fake_1000"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "02000a24",
      "metadata": {},
      "source": [
        "### Specifying the model using comparisons\n",
        "\n",
        "Splink includes a library of comparison functions at `splink.comparison_library` to make it simple to get started. These are split into two categories:\n",
        "\n",
        "1. Generic `Comparison` functions which apply a particular fuzzy matching function. For example, levenshtein distance.\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 3,
      "id": "4b7159fb",
      "metadata": {
        "execution": {
          "iopub.execute_input": "2024-07-11T13:34:23.790520Z",
          "iopub.status.busy": "2024-07-11T13:34:23.790213Z",
          "iopub.status.idle": "2024-07-11T13:34:23.818960Z",
          "shell.execute_reply": "2024-07-11T13:34:23.818252Z"
        },
        "tags": []
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Comparison 'LevenshteinAtThresholds' of \"city\".\n",
            "Similarity is assessed using the following ComparisonLevels:\n",
            "    - 'city is NULL' with SQL rule: \"city_l\" IS NULL OR \"city_r\" IS NULL\n",
            "    - 'Exact match on city' with SQL rule: \"city_l\" = \"city_r\"\n",
            "    - 'Levenshtein distance of city <= 2' with SQL rule: levenshtein(\"city_l\", \"city_r\") <= 2\n",
            "    - 'All other comparisons' with SQL rule: ELSE\n",
            "\n"
          ]
        }
      ],
      "source": [
        "import splink.comparison_library as cl\n",
        "\n",
        "city_comparison = cl.LevenshteinAtThresholds(\"city\", 2)\n",
        "print(city_comparison.get_comparison(\"duckdb\").human_readable_description)"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "f0a6cc8b",
      "metadata": {},
      "source": [
        "2. `Comparison` functions tailored for specific data types. For example, email."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 4,
      "id": "bd6143e7",
      "metadata": {
        "execution": {
          "iopub.execute_input": "2024-07-11T13:34:23.822972Z",
          "iopub.status.busy": "2024-07-11T13:34:23.822670Z",
          "iopub.status.idle": "2024-07-11T13:34:23.844781Z",
          "shell.execute_reply": "2024-07-11T13:34:23.844138Z"
        },
        "tags": []
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Comparison 'EmailComparison' of \"email\".\n",
            "Similarity is assessed using the following ComparisonLevels:\n",
            "    - 'email is NULL' with SQL rule: \"email_l\" IS NULL OR \"email_r\" IS NULL\n",
            "    - 'Exact match on email' with SQL rule: \"email_l\" = \"email_r\"\n",
            "    - 'Exact match on username' with SQL rule: NULLIF(regexp_extract(\"email_l\", '^[^@]+', 0), '') = NULLIF(regexp_extract(\"email_r\", '^[^@]+', 0), '')\n",
            "    - 'Jaro-Winkler distance of email >= 0.88' with SQL rule: jaro_winkler_similarity(\"email_l\", \"email_r\") >= 0.88\n",
            "    - 'Jaro-Winkler >0.88 on username' with SQL rule: jaro_winkler_similarity(NULLIF(regexp_extract(\"email_l\", '^[^@]+', 0), ''), NULLIF(regexp_extract(\"email_r\", '^[^@]+', 0), '')) >= 0.88\n",
            "    - 'All other comparisons' with SQL rule: ELSE\n",
            "\n"
          ]
        }
      ],
      "source": [
        "email_comparison = cl.EmailComparison(\"email\")\n",
        "print(email_comparison.get_comparison(\"duckdb\").human_readable_description)"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "47b7677a",
      "metadata": {},
      "source": [
        "## Specifying the full settings dictionary\n",
        "\n",
        "`Comparisons` are specified as part of the Splink `settings`, a Python dictionary which controls all of the configuration of a Splink model:\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 5,
      "id": "0fa0611a",
      "metadata": {
        "execution": {
          "iopub.execute_input": "2024-07-11T13:34:23.848567Z",
          "iopub.status.busy": "2024-07-11T13:34:23.848319Z",
          "iopub.status.idle": "2024-07-11T13:34:24.152927Z",
          "shell.execute_reply": "2024-07-11T13:34:24.152375Z"
        },
        "tags": []
      },
      "outputs": [],
      "source": [
        "from splink import Linker, SettingsCreator, block_on, DuckDBAPI\n",
        "\n",
        "settings = SettingsCreator(\n",
        "    link_type=\"dedupe_only\",\n",
        "    comparisons=[\n",
        "        cl.NameComparison(\"first_name\"),\n",
        "        cl.NameComparison(\"surname\"),\n",
        "        cl.LevenshteinAtThresholds(\"dob\", 1),\n",
        "        cl.ExactMatch(\"city\").configure(term_frequency_adjustments=True),\n",
        "        cl.EmailComparison(\"email\"),\n",
        "    ],\n",
        "    blocking_rules_to_generate_predictions=[\n",
        "        block_on(\"first_name\", \"city\"),\n",
        "        block_on(\"surname\"),\n",
        "\n",
        "    ],\n",
        "    retain_intermediate_calculation_columns=True,\n",
        ")\n",
        "\n",
        "linker = Linker(df, settings, db_api=DuckDBAPI())"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "657a1fb8",
      "metadata": {},
      "source": [
        "In words, this setting dictionary says:\n",
        "\n",
        "- We are performing a `dedupe_only` (the other options are `link_only`, or `link_and_dedupe`, which may be used if there are multiple input datasets).\n",
        "- When comparing records, we will use information from the `first_name`, `surname`, `dob`, `city` and `email` columns to compute a match score.\n",
        "- The `blocking_rules_to_generate_predictions` states that we will only check for duplicates amongst records where either the `first_name AND city` or `surname` is identical.\n",
        "- We have enabled [term frequency adjustments](https://moj-analytical-services.github.io/splink/topic_guides/comparisons/term-frequency.html) for the 'city' column, because some values (e.g. `London`) appear much more frequently than others.\n",
        "- We have set `retain_intermediate_calculation_columns` and `additional_columns_to_retain` to `True` so that Splink outputs additional information that helps the user understand the calculations. If they were `False`, the computations would run faster.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "afa31386",
      "metadata": {},
      "source": [
        "## Estimate the parameters of the model\n",
        "\n",
        "Now that we have specified our linkage model, we need to estimate the [`probability_two_random_records_match`](../../api_docs//settings_dict_guide.md#probability_two_random_records_match), `u`, and `m` parameters.\n",
        "\n",
        "- The `probability_two_random_records_match` parameter is the probability that two records taken at random from your input data represent a match (typically a very small number).\n",
        "\n",
        "- The `u` values are the proportion of records falling into each `ComparisonLevel` amongst truly _non-matching_ records.\n",
        "\n",
        "- The `m` values are the proportion of records falling into each `ComparisonLevel` amongst truly _matching_ records\n",
        "\n",
        "You can read more about [the theory of what these mean](https://www.robinlinacre.com/m_and_u_values/).\n",
        "\n",
        "We can estimate these parameters using unlabeled data. If we have labels, then we can estimate them even more accurately.\n",
        "\n",
        "The rationale for the approach recommended in this tutorial is documented [here](../../topic_guides/training/training_rationale.md).\n"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "c2871ac6",
      "metadata": {},
      "source": [
        "### Estimation of `probability_two_random_records_match`\n",
        "\n",
        "In some cases, the `probability_two_random_records_match` will be known. For example, if you are linking two tables of 10,000 records and expect a one-to-one match, then you should set this value to `1/10_000` in your settings instead of estimating it.\n",
        "\n",
        "More generally, this parameter is unknown and needs to be estimated.\n",
        "\n",
        "It can be estimated accurately enough for most purposes by combining a series of deterministic matching rules and a guess of the recall corresponding to those rules. For further details of the rationale behind this appraoch see [here](https://github.com/moj-analytical-services/splink/issues/462#issuecomment-1227027995).\n",
        "\n",
        "In this example, I guess that the following deterministic matching rules have a recall of about 70%. That means, between them, the rules recover 70% of all true matches.\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 6,
      "id": "cbf92120",
      "metadata": {
        "execution": {
          "iopub.execute_input": "2024-07-11T13:34:24.156645Z",
          "iopub.status.busy": "2024-07-11T13:34:24.156410Z",
          "iopub.status.idle": "2024-07-11T13:34:24.279603Z",
          "shell.execute_reply": "2024-07-11T13:34:24.279002Z"
        },
        "tags": []
      },
      "outputs": [
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "Probability two random records match is estimated to be  0.00298.\n",
            "This means that amongst all possible pairwise record comparisons, one in 335.56 are expected to match.  With 499,500 total possible comparisons, we expect a total of around 1,488.57 matching pairs\n"
          ]
        }
      ],
      "source": [
        "deterministic_rules = [\n",
        "    block_on(\"first_name\", \"dob\"),\n",
        "    \"l.first_name = r.first_name and levenshtein(r.surname, l.surname) <= 2\",\n",
        "    block_on(\"email\")\n",
        "]\n",
        "\n",
        "linker.training.estimate_probability_two_random_records_match(deterministic_rules, recall=0.7)"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "7712860b",
      "metadata": {},
      "source": [
        "### Estimation of `u` probabilities\n",
        "\n",
        "Once we have the `probability_two_random_records_match` parameter, we can estimate the `u` probabilities.\n",
        "\n",
        "We estimate `u` using the `estimate_u_using_random_sampling` method, which doesn't require any labels.\n",
        "\n",
        "It works by sampling random pairs of records, since most of these pairs are going to be non-matches. Over these non-matches we compute the distribution of `ComparisonLevel`s for each `Comparison`.\n",
        "\n",
        "For instance, for `gender`, we would find that the the gender matches 50% of the time, and mismatches 50% of the time.\n",
        "\n",
        "For `dob` on the other hand, we would find that the `dob` matches 1% of the time, has a \"one character difference\" 3% of the time, and everything else happens 96% of the time.\n",
        "\n",
        "The larger the random sample, the more accurate the predictions. You control this using the `max_pairs` parameter. For large datasets, we recommend using at least 10 million - but the higher the better and 1 billion is often appropriate for larger datasets.\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 7,
      "id": "b8d49e7a",
      "metadata": {
        "execution": {
          "iopub.execute_input": "2024-07-11T13:34:24.282770Z",
          "iopub.status.busy": "2024-07-11T13:34:24.282515Z",
          "iopub.status.idle": "2024-07-11T13:34:24.793159Z",
          "shell.execute_reply": "2024-07-11T13:34:24.792198Z"
        }
      },
      "outputs": [
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "You are using the default value for `max_pairs`, which may be too small and thus lead to inaccurate estimates for your model's u-parameters. Consider increasing to 1e8 or 1e9, which will result in more accurate estimates, but with a longer run time.\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "----- Estimating u probabilities using random sampling -----\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "\n",
            "Estimated u probabilities using random sampling\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "\n",
            "Your model is not yet fully trained. Missing estimates for:\n",
            "    - first_name (no m values are trained).\n",
            "    - surname (no m values are trained).\n",
            "    - dob (no m values are trained).\n",
            "    - city (no m values are trained).\n",
            "    - email (no m values are trained).\n"
          ]
        }
      ],
      "source": [
        "linker.training.estimate_u_using_random_sampling(max_pairs=1e6)"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "a73921b7",
      "metadata": {},
      "source": [
        "### Estimation of `m` probabilities\n",
        "\n",
        "`m` is the trickiest of the parameters to estimate, because we have to have some idea of what the true matches are.\n",
        "\n",
        "If we have labels, we can directly estimate it.\n",
        "\n",
        "If we do not have labelled data, the `m` parameters can be estimated using an iterative maximum likelihood approach called Expectation Maximisation.\n",
        "\n",
        "#### Estimating directly\n",
        "\n",
        "If we have labels, we can estimate `m` directly using the `estimate_m_from_label_column` method of the linker.\n",
        "\n",
        "For example, if the entity being matched is persons, and your input dataset(s) contain social security number, this could be used to estimate the m values for the model.\n",
        "\n",
        "Note that this column does not need to be fully populated. A common case is where a unique identifier such as social security number is only partially populated.\n",
        "\n",
        "For example (in this tutorial we don't have labels, so we're not actually going to use this):\n",
        "\n",
        "```python\n",
        "linker.training.estimate_m_from_label_column(\"social_security_number\")\n",
        "```\n",
        "\n",
        "#### Estimating with Expectation Maximisation\n",
        "\n",
        "This algorithm estimates the `m` values by generating pairwise record comparisons, and using them to maximise a likelihood function.\n",
        "\n",
        "Each estimation pass requires the user to configure an estimation blocking rule to reduce the number of record comparisons generated to a manageable level.\n",
        "\n",
        "In our first estimation pass, we block on `first_name` and `surname`, meaning we will generate all record comparisons that have `first_name` and `surname` exactly equal.\n",
        "\n",
        "Recall we are trying to estimate the `m` values of the model, i.e. proportion of records falling into each `ComparisonLevel` amongst truly matching records.\n",
        "\n",
        "This means that, in this training session, we cannot estimate parameter estimates for the `first_name` or `surname` columns, since they will be equal for all the comparisons we do.\n",
        "\n",
        "We can, however, estimate parameter estimates for all of the other columns. The output messages produced by Splink confirm this.\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 8,
      "id": "098f0a40",
      "metadata": {
        "execution": {
          "iopub.execute_input": "2024-07-11T13:34:24.796381Z",
          "iopub.status.busy": "2024-07-11T13:34:24.796116Z",
          "iopub.status.idle": "2024-07-11T13:34:25.542726Z",
          "shell.execute_reply": "2024-07-11T13:34:25.541946Z"
        }
      },
      "outputs": [
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "\n",
            "----- Starting EM training session -----\n",
            "\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "Estimating the m probabilities of the model by blocking on:\n",
            "(l.\"first_name\" = r.\"first_name\") AND (l.\"surname\" = r.\"surname\")\n",
            "\n",
            "Parameter estimates will be made for the following comparison(s):\n",
            "    - dob\n",
            "    - city\n",
            "    - email\n",
            "\n",
            "Parameter estimates cannot be made for the following comparison(s) since they are used in the blocking rules: \n",
            "    - first_name\n",
            "    - surname\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "WARNING:\n",
            "Level Jaro-Winkler >0.88 on username on comparison email not observed in dataset, unable to train m value\n",
            "\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "Iteration 1: Largest change in params was -0.521 in the m_probability of dob, level `Exact match on dob`\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "Iteration 2: Largest change in params was 0.0516 in probability_two_random_records_match\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "Iteration 3: Largest change in params was 0.0183 in probability_two_random_records_match\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "Iteration 4: Largest change in params was 0.00744 in probability_two_random_records_match\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "Iteration 5: Largest change in params was 0.00349 in probability_two_random_records_match\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "Iteration 6: Largest change in params was 0.00183 in probability_two_random_records_match\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "Iteration 7: Largest change in params was 0.00103 in probability_two_random_records_match\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "Iteration 8: Largest change in params was 0.000607 in probability_two_random_records_match\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "Iteration 9: Largest change in params was 0.000367 in probability_two_random_records_match\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "Iteration 10: Largest change in params was 0.000226 in probability_two_random_records_match\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "Iteration 11: Largest change in params was 0.00014 in probability_two_random_records_match\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "Iteration 12: Largest change in params was 8.73e-05 in probability_two_random_records_match\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "\n",
            "EM converged after 12 iterations\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "m probability not trained for email - Jaro-Winkler >0.88 on username (comparison vector value: 1). This usually means the comparison level was never observed in the training data.\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "\n",
            "Your model is not yet fully trained. Missing estimates for:\n",
            "    - first_name (no m values are trained).\n",
            "    - surname (no m values are trained).\n",
            "    - email (some m values are not trained).\n"
          ]
        }
      ],
      "source": [
        "training_blocking_rule = block_on(\"first_name\", \"surname\")\n",
        "training_session_fname_sname = (\n",
        "    linker.training.estimate_parameters_using_expectation_maximisation(training_blocking_rule)\n",
        ")"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "92bd4a31",
      "metadata": {},
      "source": [
        "In a second estimation pass, we block on dob. This allows us to estimate parameters for the `first_name` and `surname` comparisons.\n",
        "\n",
        "Between the two estimation passes, we now have parameter estimates for all comparisons.\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 9,
      "id": "ac8d3264",
      "metadata": {
        "execution": {
          "iopub.execute_input": "2024-07-11T13:34:25.546223Z",
          "iopub.status.busy": "2024-07-11T13:34:25.545981Z",
          "iopub.status.idle": "2024-07-11T13:34:26.550128Z",
          "shell.execute_reply": "2024-07-11T13:34:26.549700Z"
        }
      },
      "outputs": [
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "\n",
            "----- Starting EM training session -----\n",
            "\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "Estimating the m probabilities of the model by blocking on:\n",
            "l.\"dob\" = r.\"dob\"\n",
            "\n",
            "Parameter estimates will be made for the following comparison(s):\n",
            "    - first_name\n",
            "    - surname\n",
            "    - city\n",
            "    - email\n",
            "\n",
            "Parameter estimates cannot be made for the following comparison(s) since they are used in the blocking rules: \n",
            "    - dob\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "WARNING:\n",
            "Level Jaro-Winkler >0.88 on username on comparison email not observed in dataset, unable to train m value\n",
            "\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "Iteration 1: Largest change in params was -0.407 in the m_probability of surname, level `Exact match on surname`\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "Iteration 2: Largest change in params was 0.0929 in probability_two_random_records_match\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "Iteration 3: Largest change in params was 0.0548 in the m_probability of first_name, level `All other comparisons`\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "Iteration 4: Largest change in params was 0.0186 in probability_two_random_records_match\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "Iteration 5: Largest change in params was 0.00758 in probability_two_random_records_match\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "Iteration 6: Largest change in params was 0.00339 in probability_two_random_records_match\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "Iteration 7: Largest change in params was 0.0016 in probability_two_random_records_match\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "Iteration 8: Largest change in params was 0.000773 in probability_two_random_records_match\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "Iteration 9: Largest change in params was 0.000379 in probability_two_random_records_match\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "Iteration 10: Largest change in params was 0.000189 in probability_two_random_records_match\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "Iteration 11: Largest change in params was 9.68e-05 in probability_two_random_records_match\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "\n",
            "EM converged after 11 iterations\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "m probability not trained for email - Jaro-Winkler >0.88 on username (comparison vector value: 1). This usually means the comparison level was never observed in the training data.\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "\n",
            "Your model is not yet fully trained. Missing estimates for:\n",
            "    - email (some m values are not trained).\n"
          ]
        }
      ],
      "source": [
        "training_blocking_rule = block_on(\"dob\")\n",
        "training_session_dob = linker.training.estimate_parameters_using_expectation_maximisation(\n",
        "    training_blocking_rule\n",
        ")"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "efdb0c5f",
      "metadata": {},
      "source": [
        "Note that Splink includes other algorithms for estimating m and u values, which are documented [here](https://moj-analytical-services.github.io/splink/api_docs/training.html).\n"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "38355535",
      "metadata": {},
      "source": [
        "## Visualising model parameters\n",
        "\n",
        "Splink can generate a number of charts to help you understand your model. For an introduction to these charts and how to interpret them, please see [this](https://www.youtube.com/watch?v=msz3T741KQI&t=507s) video.\n",
        "\n",
        "The final estimated match weights can be viewed in the match weights chart:\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 10,
      "id": "3a1e15cc",
      "metadata": {
        "execution": {
          "iopub.execute_input": "2024-07-11T13:34:26.554693Z",
          "iopub.status.busy": "2024-07-11T13:34:26.554427Z",
          "iopub.status.idle": "2024-07-11T13:34:26.997140Z",
          "shell.execute_reply": "2024-07-11T13:34:26.996455Z"
        }
      },
      "outputs": [
        {
          "data": {
            "text/html": [
              "\n",
              "<style>\n",
              "  #altair-viz-8d889a1656cb4cfc8041a758cd668ee1.vega-embed {\n",
              "    width: 100%;\n",
              "    display: flex;\n",
              "  }\n",
              "\n",
              "  #altair-viz-8d889a1656cb4cfc8041a758cd668ee1.vega-embed details,\n",
              "  #altair-viz-8d889a1656cb4cfc8041a758cd668ee1.vega-embed details summary {\n",
              "    position: relative;\n",
              "  }\n",
              "</style>\n",
              "<div id=\"altair-viz-8d889a1656cb4cfc8041a758cd668ee1\"></div>\n",
              "<script type=\"text/javascript\">\n",
              "  var VEGA_DEBUG = (typeof VEGA_DEBUG == \"undefined\") ? {} : VEGA_DEBUG;\n",
              "  (function(spec, embedOpt){\n",
              "    let outputDiv = document.currentScript.previousElementSibling;\n",
              "    if (outputDiv.id !== \"altair-viz-8d889a1656cb4cfc8041a758cd668ee1\") {\n",
              "      outputDiv = document.getElementById(\"altair-viz-8d889a1656cb4cfc8041a758cd668ee1\");\n",
              "    }\n",
              "    const paths = {\n",
              "      \"vega\": \"https://cdn.jsdelivr.net/npm/vega@5?noext\",\n",
              "      \"vega-lib\": \"https://cdn.jsdelivr.net/npm/vega-lib?noext\",\n",
              "      \"vega-lite\": \"https://cdn.jsdelivr.net/npm/vega-lite@5.17.0?noext\",\n",
              "      \"vega-embed\": \"https://cdn.jsdelivr.net/npm/vega-embed@6?noext\",\n",
              "    };\n",
              "\n",
              "    function maybeLoadScript(lib, version) {\n",
              "      var key = `${lib.replace(\"-\", \"\")}_version`;\n",
              "      return (VEGA_DEBUG[key] == version) ?\n",
              "        Promise.resolve(paths[lib]) :\n",
              "        new Promise(function(resolve, reject) {\n",
              "          var s = document.createElement('script');\n",
              "          document.getElementsByTagName(\"head\")[0].appendChild(s);\n",
              "          s.async = true;\n",
              "          s.onload = () => {\n",
              "            VEGA_DEBUG[key] = version;\n",
              "            return resolve(paths[lib]);\n",
              "          };\n",
              "          s.onerror = () => reject(`Error loading script: ${paths[lib]}`);\n",
              "          s.src = paths[lib];\n",
              "        });\n",
              "    }\n",
              "\n",
              "    function showError(err) {\n",
              "      outputDiv.innerHTML = `<div class=\"error\" style=\"color:red;\">${err}</div>`;\n",
              "      throw err;\n",
              "    }\n",
              "\n",
              "    function displayChart(vegaEmbed) {\n",
              "      vegaEmbed(outputDiv, spec, embedOpt)\n",
              "        .catch(err => showError(`Javascript Error: ${err.message}<br>This usually means there's a typo in your chart specification. See the javascript console for the full traceback.`));\n",
              "    }\n",
              "\n",
              "    if(typeof define === \"function\" && define.amd) {\n",
              "      requirejs.config({paths});\n",
              "      require([\"vega-embed\"], displayChart, err => showError(`Error loading script: ${err.message}`));\n",
              "    } else {\n",
              "      maybeLoadScript(\"vega\", \"5\")\n",
              "        .then(() => maybeLoadScript(\"vega-lite\", \"5.17.0\"))\n",
              "        .then(() => maybeLoadScript(\"vega-embed\", \"6\"))\n",
              "        .catch(showError)\n",
              "        .then(() => displayChart(vegaEmbed));\n",
              "    }\n",
              "  })({\"config\": {\"view\": {\"continuousWidth\": 300, \"continuousHeight\": 300, \"discreteHeight\": 60, \"discreteWidth\": 400}, \"header\": {\"title\": null}, \"mark\": {\"tooltip\": null}, \"title\": {\"anchor\": \"middle\"}}, \"vconcat\": [{\"mark\": {\"type\": \"bar\", \"clip\": true, \"height\": 15}, \"encoding\": {\"color\": {\"field\": \"log2_bayes_factor\", \"scale\": {\"domain\": [-10, 0, 10], \"interpolate\": \"lab\", \"range\": [\"red\", \"#bbbbbb\", \"green\"]}, \"title\": \"Match weight\", \"type\": \"quantitative\"}, \"tooltip\": [{\"field\": \"comparison_name\", \"title\": \"Comparison name\", \"type\": \"nominal\"}, {\"field\": \"probability_two_random_records_match\", \"format\": \".4f\", \"title\": \"Probability two random records match\", \"type\": \"nominal\"}, {\"field\": \"log2_bayes_factor\", \"format\": \",.4f\", \"title\": \"Equivalent match weight\", \"type\": \"quantitative\"}, {\"field\": \"bayes_factor_description\", \"title\": \"Match weight description\", \"type\": \"nominal\"}], \"x\": {\"axis\": {\"domain\": false, \"gridColor\": {\"condition\": {\"test\": \"abs(datum.value / 10)  <= 1 & datum.value % 10 === 0\", \"value\": \"#aaa\"}, \"value\": \"#ddd\"}, \"gridDash\": {\"condition\": {\"test\": \"abs(datum.value / 10) == 1\", \"value\": [3]}, \"value\": null}, \"gridWidth\": {\"condition\": {\"test\": \"abs(datum.value / 10)  <= 1 & datum.value % 10 === 0\", \"value\": 2}, \"value\": 1}, \"labels\": false, \"ticks\": false, \"title\": \"\"}, \"field\": \"log2_bayes_factor\", \"scale\": {\"domain\": [-9, 9]}, \"type\": \"quantitative\"}, \"y\": {\"axis\": {\"title\": \"Prior (starting) match weight\", \"titleAlign\": \"right\", \"titleAngle\": 0, \"titleFontWeight\": \"normal\"}, \"field\": \"label_for_charts\", \"sort\": {\"field\": \"comparison_vector_value\", \"order\": \"descending\"}, \"type\": \"nominal\"}}, \"height\": 20, \"transform\": [{\"filter\": \"(datum.comparison_name == 'probability_two_random_records_match')\"}]}, {\"mark\": {\"type\": \"bar\", \"clip\": true}, \"encoding\": {\"color\": {\"field\": \"log2_bayes_factor\", \"scale\": {\"domain\": [-10, 0, 10], \"interpolate\": \"lab\", \"range\": [\"red\", \"#bbbbbb\", \"green\"]}, \"title\": \"Match weight\", \"type\": \"quantitative\"}, \"row\": {\"field\": \"comparison_name\", \"header\": {\"labelAlign\": \"left\", \"labelAnchor\": \"middle\", \"labelAngle\": 0}, \"sort\": {\"field\": \"comparison_sort_order\"}, \"type\": \"nominal\"}, \"tooltip\": [{\"field\": \"comparison_name\", \"title\": \"Comparison name\", \"type\": \"nominal\"}, {\"field\": \"label_for_charts\", \"title\": \"Label\", \"type\": \"ordinal\"}, {\"field\": \"sql_condition\", \"title\": \"SQL condition\", \"type\": \"nominal\"}, {\"field\": \"m_probability\", \"format\": \".4f\", \"title\": \"M probability\", \"type\": \"quantitative\"}, {\"field\": \"u_probability\", \"format\": \".4f\", \"title\": \"U probability\", \"type\": \"quantitative\"}, {\"field\": \"bayes_factor\", \"format\": \",.4f\", \"title\": \"Bayes factor = m/u\", \"type\": \"quantitative\"}, {\"field\": \"log2_bayes_factor\", \"format\": \",.4f\", \"title\": \"Match weight = log2(m/u)\", \"type\": \"quantitative\"}, {\"field\": \"bayes_factor_description\", \"title\": \"Match weight description\", \"type\": \"nominal\"}], \"x\": {\"axis\": {\"gridColor\": {\"condition\": {\"test\": \"abs(datum.value / 10)  <= 1 & datum.value % 10 === 0\", \"value\": \"#aaa\"}, \"value\": \"#ddd\"}, \"gridDash\": {\"condition\": {\"test\": \"abs(datum.value / 10) == 1\", \"value\": [3]}, \"value\": null}, \"gridWidth\": {\"condition\": {\"test\": \"abs(datum.value / 10)  <= 1 & datum.value % 10 === 0\", \"value\": 2}, \"value\": 1}, \"title\": \"Comparison level match weight = log2(m/u)\"}, \"field\": \"log2_bayes_factor\", \"scale\": {\"domain\": [-9, 9]}, \"type\": \"quantitative\"}, \"y\": {\"axis\": {\"title\": null}, \"field\": \"label_for_charts\", \"sort\": {\"field\": \"comparison_vector_value\", \"order\": \"descending\"}, \"type\": \"nominal\"}}, \"height\": {\"step\": 12}, \"resolve\": {\"axis\": {\"y\": \"independent\"}, \"scale\": {\"y\": \"independent\"}}, \"transform\": [{\"filter\": \"(datum.comparison_name != 'probability_two_random_records_match')\"}]}], \"data\": {\"name\": \"data-4085515acfc9ce98d80c83805f954c87\"}, \"params\": [{\"name\": \"mouse_zoom\", \"select\": {\"type\": \"interval\", \"encodings\": [\"x\"]}, \"bind\": \"scales\", \"views\": []}], \"resolve\": {\"axis\": {\"y\": \"independent\"}, \"scale\": {\"y\": \"independent\"}}, \"title\": {\"text\": \"Model parameters (components of final match weight)\", \"subtitle\": \"Use mousewheel to zoom\"}, \"$schema\": \"https://vega.github.io/schema/vega-lite/v5.9.3.json\", \"datasets\": {\"data-4085515acfc9ce98d80c83805f954c87\": [{\"comparison_name\": \"probability_two_random_records_match\", \"sql_condition\": null, \"label_for_charts\": \"\", \"m_probability\": null, \"u_probability\": null, \"m_probability_description\": null, \"u_probability_description\": null, \"has_tf_adjustments\": false, \"tf_adjustment_column\": null, \"tf_adjustment_weight\": null, \"is_null_level\": false, \"bayes_factor\": 0.002989030659078392, \"log2_bayes_factor\": -8.386106589561935, \"comparison_vector_value\": 0, \"max_comparison_vector_value\": 0, \"bayes_factor_description\": \"The probability that two random records drawn at random match is 0.003 or one in  335.6 records.This is equivalent to a starting match weight of -8.386.\", \"probability_two_random_records_match\": 0.00298012298012298, \"comparison_sort_order\": -1}, {\"comparison_name\": \"first_name\", \"sql_condition\": \"\\\"first_name_l\\\" = \\\"first_name_r\\\"\", \"label_for_charts\": \"Exact match on first_name\", \"m_probability\": 0.4914209493176376, \"u_probability\": 0.0057935713975033705, \"m_probability_description\": \"Amongst matching record comparisons, 49.14% of records are in the exact match on first_name comparison level\", \"u_probability_description\": \"Amongst non-matching record comparisons, 0.58% of records are in the exact match on first_name comparison level\", \"has_tf_adjustments\": true, \"tf_adjustment_column\": \"first_name\", \"tf_adjustment_weight\": 1.0, \"is_null_level\": false, \"bayes_factor\": 84.82176460782136, \"log2_bayes_factor\": 6.406362591550452, \"comparison_vector_value\": 4, \"max_comparison_vector_value\": 4, \"bayes_factor_description\": \"If comparison level is `exact match on first_name` then comparison is 84.82 times more likely to be a match\", \"probability_two_random_records_match\": 0.00298012298012298, \"comparison_sort_order\": 0}, {\"comparison_name\": \"first_name\", \"sql_condition\": \"jaro_winkler_similarity(\\\"first_name_l\\\", \\\"first_name_r\\\") >= 0.92\", \"label_for_charts\": \"Jaro-Winkler distance of first_name >= 0.92\", \"m_probability\": 0.15176057384758346, \"u_probability\": 0.0023429457903817435, \"m_probability_description\": \"Amongst matching record comparisons, 15.18% of records are in the jaro-winkler distance of first_name >= 0.92 comparison level\", \"u_probability_description\": \"Amongst non-matching record comparisons, 0.23% of records are in the jaro-winkler distance of first_name >= 0.92 comparison level\", \"has_tf_adjustments\": false, \"tf_adjustment_column\": null, \"tf_adjustment_weight\": 1.0, \"is_null_level\": false, \"bayes_factor\": 64.77340383656791, \"log2_bayes_factor\": 6.017329654456136, \"comparison_vector_value\": 3, \"max_comparison_vector_value\": 4, \"bayes_factor_description\": \"If comparison level is `jaro-winkler distance of first_name >= 0.92` then comparison is 64.77 times more likely to be a match\", \"probability_two_random_records_match\": 0.00298012298012298, \"comparison_sort_order\": 0}, {\"comparison_name\": \"first_name\", \"sql_condition\": \"jaro_winkler_similarity(\\\"first_name_l\\\", \\\"first_name_r\\\") >= 0.88\", \"label_for_charts\": \"Jaro-Winkler distance of first_name >= 0.88\", \"m_probability\": 0.07406496776118927, \"u_probability\": 0.0015484319951285285, \"m_probability_description\": \"Amongst matching record comparisons, 7.41% of records are in the jaro-winkler distance of first_name >= 0.88 comparison level\", \"u_probability_description\": \"Amongst non-matching record comparisons, 0.15% of records are in the jaro-winkler distance of first_name >= 0.88 comparison level\", \"has_tf_adjustments\": false, \"tf_adjustment_column\": null, \"tf_adjustment_weight\": 1.0, \"is_null_level\": false, \"bayes_factor\": 47.83223802801974, \"log2_bayes_factor\": 5.579911390213005, \"comparison_vector_value\": 2, \"max_comparison_vector_value\": 4, \"bayes_factor_description\": \"If comparison level is `jaro-winkler distance of first_name >= 0.88` then comparison is 47.83 times more likely to be a match\", \"probability_two_random_records_match\": 0.00298012298012298, \"comparison_sort_order\": 0}, {\"comparison_name\": \"first_name\", \"sql_condition\": \"jaro_winkler_similarity(\\\"first_name_l\\\", \\\"first_name_r\\\") >= 0.7\", \"label_for_charts\": \"Jaro-Winkler distance of first_name >= 0.7\", \"m_probability\": 0.0790861077150486, \"u_probability\": 0.018934945558406913, \"m_probability_description\": \"Amongst matching record comparisons, 7.91% of records are in the jaro-winkler distance of first_name >= 0.7 comparison level\", \"u_probability_description\": \"Amongst non-matching record comparisons, 1.89% of records are in the jaro-winkler distance of first_name >= 0.7 comparison level\", \"has_tf_adjustments\": false, \"tf_adjustment_column\": null, \"tf_adjustment_weight\": 1.0, \"is_null_level\": false, \"bayes_factor\": 4.176727494203711, \"log2_bayes_factor\": 2.0623730195874392, \"comparison_vector_value\": 1, \"max_comparison_vector_value\": 4, \"bayes_factor_description\": \"If comparison level is `jaro-winkler distance of first_name >= 0.7` then comparison is 4.18 times more likely to be a match\", \"probability_two_random_records_match\": 0.00298012298012298, \"comparison_sort_order\": 0}, {\"comparison_name\": \"first_name\", \"sql_condition\": \"ELSE\", \"label_for_charts\": \"All other comparisons\", \"m_probability\": 0.2036674013585411, \"u_probability\": 0.9713801052585794, \"m_probability_description\": \"Amongst matching record comparisons, 20.37% of records are in the all other comparisons comparison level\", \"u_probability_description\": \"Amongst non-matching record comparisons, 97.14% of records are in the all other comparisons comparison level\", \"has_tf_adjustments\": false, \"tf_adjustment_column\": null, \"tf_adjustment_weight\": 1.0, \"is_null_level\": false, \"bayes_factor\": 0.2096680797310804, \"log2_bayes_factor\": -2.25382085538399, \"comparison_vector_value\": 0, \"max_comparison_vector_value\": 4, \"bayes_factor_description\": \"If comparison level is `all other comparisons` then comparison is  4.77 times less likely to be a match\", \"probability_two_random_records_match\": 0.00298012298012298, \"comparison_sort_order\": 0}, {\"comparison_name\": \"surname\", \"sql_condition\": \"\\\"surname_l\\\" = \\\"surname_r\\\"\", \"label_for_charts\": \"Exact match on surname\", \"m_probability\": 0.43457460622893695, \"u_probability\": 0.004889975550122249, \"m_probability_description\": \"Amongst matching record comparisons, 43.46% of records are in the exact match on surname comparison level\", \"u_probability_description\": \"Amongst non-matching record comparisons, 0.49% of records are in the exact match on surname comparison level\", \"has_tf_adjustments\": true, \"tf_adjustment_column\": \"surname\", \"tf_adjustment_weight\": 1.0, \"is_null_level\": false, \"bayes_factor\": 88.87050697381761, \"log2_bayes_factor\": 6.47363281313057, \"comparison_vector_value\": 4, \"max_comparison_vector_value\": 4, \"bayes_factor_description\": \"If comparison level is `exact match on surname` then comparison is 88.87 times more likely to be a match\", \"probability_two_random_records_match\": 0.00298012298012298, \"comparison_sort_order\": 1}, {\"comparison_name\": \"surname\", \"sql_condition\": \"jaro_winkler_similarity(\\\"surname_l\\\", \\\"surname_r\\\") >= 0.92\", \"label_for_charts\": \"Jaro-Winkler distance of surname >= 0.92\", \"m_probability\": 0.22529103510053114, \"u_probability\": 0.00283905173880724, \"m_probability_description\": \"Amongst matching record comparisons, 22.53% of records are in the jaro-winkler distance of surname >= 0.92 comparison level\", \"u_probability_description\": \"Amongst non-matching record comparisons, 0.28% of records are in the jaro-winkler distance of surname >= 0.92 comparison level\", \"has_tf_adjustments\": false, \"tf_adjustment_column\": null, \"tf_adjustment_weight\": 1.0, \"is_null_level\": false, \"bayes_factor\": 79.35432525621452, \"log2_bayes_factor\": 6.3102369550698105, \"comparison_vector_value\": 3, \"max_comparison_vector_value\": 4, \"bayes_factor_description\": \"If comparison level is `jaro-winkler distance of surname >= 0.92` then comparison is 79.35 times more likely to be a match\", \"probability_two_random_records_match\": 0.00298012298012298, \"comparison_sort_order\": 1}, {\"comparison_name\": \"surname\", \"sql_condition\": \"jaro_winkler_similarity(\\\"surname_l\\\", \\\"surname_r\\\") >= 0.88\", \"label_for_charts\": \"Jaro-Winkler distance of surname >= 0.88\", \"m_probability\": 0.04432221456939868, \"u_probability\": 0.0011314412292407403, \"m_probability_description\": \"Amongst matching record comparisons, 4.43% of records are in the jaro-winkler distance of surname >= 0.88 comparison level\", \"u_probability_description\": \"Amongst non-matching record comparisons, 0.11% of records are in the jaro-winkler distance of surname >= 0.88 comparison level\", \"has_tf_adjustments\": false, \"tf_adjustment_column\": null, \"tf_adjustment_weight\": 1.0, \"is_null_level\": false, \"bayes_factor\": 39.17323624413204, \"log2_bayes_factor\": 5.291796414406481, \"comparison_vector_value\": 2, \"max_comparison_vector_value\": 4, \"bayes_factor_description\": \"If comparison level is `jaro-winkler distance of surname >= 0.88` then comparison is 39.17 times more likely to be a match\", \"probability_two_random_records_match\": 0.00298012298012298, \"comparison_sort_order\": 1}, {\"comparison_name\": \"surname\", \"sql_condition\": \"jaro_winkler_similarity(\\\"surname_l\\\", \\\"surname_r\\\") >= 0.7\", \"label_for_charts\": \"Jaro-Winkler distance of surname >= 0.7\", \"m_probability\": 0.07784087870954867, \"u_probability\": 0.014729633311540402, \"m_probability_description\": \"Amongst matching record comparisons, 7.78% of records are in the jaro-winkler distance of surname >= 0.7 comparison level\", \"u_probability_description\": \"Amongst non-matching record comparisons, 1.47% of records are in the jaro-winkler distance of surname >= 0.7 comparison level\", \"has_tf_adjustments\": false, \"tf_adjustment_column\": null, \"tf_adjustment_weight\": 1.0, \"is_null_level\": false, \"bayes_factor\": 5.28464470656997, \"log2_bayes_factor\": 2.4018064806335238, \"comparison_vector_value\": 1, \"max_comparison_vector_value\": 4, \"bayes_factor_description\": \"If comparison level is `jaro-winkler distance of surname >= 0.7` then comparison is 5.28 times more likely to be a match\", \"probability_two_random_records_match\": 0.00298012298012298, \"comparison_sort_order\": 1}, {\"comparison_name\": \"surname\", \"sql_condition\": \"ELSE\", \"label_for_charts\": \"All other comparisons\", \"m_probability\": 0.2179712653915845, \"u_probability\": 0.9764098981702893, \"m_probability_description\": \"Amongst matching record comparisons, 21.80% of records are in the all other comparisons comparison level\", \"u_probability_description\": \"Amongst non-matching record comparisons, 97.64% of records are in the all other comparisons comparison level\", \"has_tf_adjustments\": false, \"tf_adjustment_column\": null, \"tf_adjustment_weight\": 1.0, \"is_null_level\": false, \"bayes_factor\": 0.22323745980048387, \"log2_bayes_factor\": -2.1633489595936646, \"comparison_vector_value\": 0, \"max_comparison_vector_value\": 4, \"bayes_factor_description\": \"If comparison level is `all other comparisons` then comparison is  4.48 times less likely to be a match\", \"probability_two_random_records_match\": 0.00298012298012298, \"comparison_sort_order\": 1}, {\"comparison_name\": \"dob\", \"sql_condition\": \"\\\"dob_l\\\" = \\\"dob_r\\\"\", \"label_for_charts\": \"Exact match on dob\", \"m_probability\": 0.3914216652882992, \"u_probability\": 0.0017477477477477479, \"m_probability_description\": \"Amongst matching record comparisons, 39.14% of records are in the exact match on dob comparison level\", \"u_probability_description\": \"Amongst non-matching record comparisons, 0.17% of records are in the exact match on dob comparison level\", \"has_tf_adjustments\": false, \"tf_adjustment_column\": null, \"tf_adjustment_weight\": 1.0, \"is_null_level\": false, \"bayes_factor\": 223.95775694330518, \"log2_bayes_factor\": 7.807082825648326, \"comparison_vector_value\": 2, \"max_comparison_vector_value\": 2, \"bayes_factor_description\": \"If comparison level is `exact match on dob` then comparison is 223.96 times more likely to be a match\", \"probability_two_random_records_match\": 0.00298012298012298, \"comparison_sort_order\": 2}, {\"comparison_name\": \"dob\", \"sql_condition\": \"levenshtein(\\\"dob_l\\\", \\\"dob_r\\\") <= 1\", \"label_for_charts\": \"Levenshtein distance of dob <= 1\", \"m_probability\": 0.14937817941895074, \"u_probability\": 0.0016016016016016017, \"m_probability_description\": \"Amongst matching record comparisons, 14.94% of records are in the levenshtein distance of dob <= 1 comparison level\", \"u_probability_description\": \"Amongst non-matching record comparisons, 0.16% of records are in the levenshtein distance of dob <= 1 comparison level\", \"has_tf_adjustments\": false, \"tf_adjustment_column\": null, \"tf_adjustment_weight\": 1.0, \"is_null_level\": false, \"bayes_factor\": 93.26800077470736, \"log2_bayes_factor\": 6.543310288053254, \"comparison_vector_value\": 1, \"max_comparison_vector_value\": 2, \"bayes_factor_description\": \"If comparison level is `levenshtein distance of dob <= 1` then comparison is 93.27 times more likely to be a match\", \"probability_two_random_records_match\": 0.00298012298012298, \"comparison_sort_order\": 2}, {\"comparison_name\": \"dob\", \"sql_condition\": \"ELSE\", \"label_for_charts\": \"All other comparisons\", \"m_probability\": 0.45920015529275016, \"u_probability\": 0.9966506506506506, \"m_probability_description\": \"Amongst matching record comparisons, 45.92% of records are in the all other comparisons comparison level\", \"u_probability_description\": \"Amongst non-matching record comparisons, 99.67% of records are in the all other comparisons comparison level\", \"has_tf_adjustments\": false, \"tf_adjustment_column\": null, \"tf_adjustment_weight\": 1.0, \"is_null_level\": false, \"bayes_factor\": 0.4607433457179476, \"log2_bayes_factor\": -1.1179647649923983, \"comparison_vector_value\": 0, \"max_comparison_vector_value\": 2, \"bayes_factor_description\": \"If comparison level is `all other comparisons` then comparison is  2.17 times less likely to be a match\", \"probability_two_random_records_match\": 0.00298012298012298, \"comparison_sort_order\": 2}, {\"comparison_name\": \"city\", \"sql_condition\": \"\\\"city_l\\\" = \\\"city_r\\\"\", \"label_for_charts\": \"Exact match on city\", \"m_probability\": 0.562574722357491, \"u_probability\": 0.0551475711801453, \"m_probability_description\": \"Amongst matching record comparisons, 56.26% of records are in the exact match on city comparison level\", \"u_probability_description\": \"Amongst non-matching record comparisons, 5.51% of records are in the exact match on city comparison level\", \"has_tf_adjustments\": true, \"tf_adjustment_column\": \"city\", \"tf_adjustment_weight\": 1.0, \"is_null_level\": false, \"bayes_factor\": 10.201260188228089, \"log2_bayes_factor\": 3.350675477967205, \"comparison_vector_value\": 1, \"max_comparison_vector_value\": 1, \"bayes_factor_description\": \"If comparison level is `exact match on city` then comparison is 10.20 times more likely to be a match\", \"probability_two_random_records_match\": 0.00298012298012298, \"comparison_sort_order\": 3}, {\"comparison_name\": \"city\", \"sql_condition\": \"ELSE\", \"label_for_charts\": \"All other comparisons\", \"m_probability\": 0.4374252776425091, \"u_probability\": 0.9448524288198547, \"m_probability_description\": \"Amongst matching record comparisons, 43.74% of records are in the all other comparisons comparison level\", \"u_probability_description\": \"Amongst non-matching record comparisons, 94.49% of records are in the all other comparisons comparison level\", \"has_tf_adjustments\": false, \"tf_adjustment_column\": null, \"tf_adjustment_weight\": 1.0, \"is_null_level\": false, \"bayes_factor\": 0.46295618691361645, \"log2_bayes_factor\": -1.111052428212497, \"comparison_vector_value\": 0, \"max_comparison_vector_value\": 1, \"bayes_factor_description\": \"If comparison level is `all other comparisons` then comparison is  2.16 times less likely to be a match\", \"probability_two_random_records_match\": 0.00298012298012298, \"comparison_sort_order\": 3}, {\"comparison_name\": \"email\", \"sql_condition\": \"\\\"email_l\\\" = \\\"email_r\\\"\", \"label_for_charts\": \"Exact match on email\", \"m_probability\": 0.5529665836585227, \"u_probability\": 0.0021938713143283602, \"m_probability_description\": \"Amongst matching record comparisons, 55.30% of records are in the exact match on email comparison level\", \"u_probability_description\": \"Amongst non-matching record comparisons, 0.22% of records are in the exact match on email comparison level\", \"has_tf_adjustments\": true, \"tf_adjustment_column\": \"email\", \"tf_adjustment_weight\": 1.0, \"is_null_level\": false, \"bayes_factor\": 252.0506011665547, \"log2_bayes_factor\": 7.977569585101858, \"comparison_vector_value\": 4, \"max_comparison_vector_value\": 4, \"bayes_factor_description\": \"If comparison level is `exact match on email` then comparison is 252.05 times more likely to be a match\", \"probability_two_random_records_match\": 0.00298012298012298, \"comparison_sort_order\": 4}, {\"comparison_name\": \"email\", \"sql_condition\": \"NULLIF(regexp_extract(\\\"email_l\\\", '^[^@]+', 0), '') = NULLIF(regexp_extract(\\\"email_r\\\", '^[^@]+', 0), '')\", \"label_for_charts\": \"Exact match on username\", \"m_probability\": 0.2208741262673713, \"u_probability\": 0.0010390328952024346, \"m_probability_description\": \"Amongst matching record comparisons, 22.09% of records are in the exact match on username comparison level\", \"u_probability_description\": \"Amongst non-matching record comparisons, 0.10% of records are in the exact match on username comparison level\", \"has_tf_adjustments\": false, \"tf_adjustment_column\": null, \"tf_adjustment_weight\": 1.0, \"is_null_level\": false, \"bayes_factor\": 212.57664438462118, \"log2_bayes_factor\": 7.731839287671784, \"comparison_vector_value\": 3, \"max_comparison_vector_value\": 4, \"bayes_factor_description\": \"If comparison level is `exact match on username` then comparison is 212.58 times more likely to be a match\", \"probability_two_random_records_match\": 0.00298012298012298, \"comparison_sort_order\": 4}, {\"comparison_name\": \"email\", \"sql_condition\": \"jaro_winkler_similarity(\\\"email_l\\\", \\\"email_r\\\") >= 0.88\", \"label_for_charts\": \"Jaro-Winkler distance of email >= 0.88\", \"m_probability\": 0.2141299946482688, \"u_probability\": 0.0009135769109519858, \"m_probability_description\": \"Amongst matching record comparisons, 21.41% of records are in the jaro-winkler distance of email >= 0.88 comparison level\", \"u_probability_description\": \"Amongst non-matching record comparisons, 0.09% of records are in the jaro-winkler distance of email >= 0.88 comparison level\", \"has_tf_adjustments\": false, \"tf_adjustment_column\": null, \"tf_adjustment_weight\": 1.0, \"is_null_level\": false, \"bayes_factor\": 234.38639055045326, \"log2_bayes_factor\": 7.872744993087004, \"comparison_vector_value\": 2, \"max_comparison_vector_value\": 4, \"bayes_factor_description\": \"If comparison level is `jaro-winkler distance of email >= 0.88` then comparison is 234.39 times more likely to be a match\", \"probability_two_random_records_match\": 0.00298012298012298, \"comparison_sort_order\": 4}, {\"comparison_name\": \"email\", \"sql_condition\": \"jaro_winkler_similarity(NULLIF(regexp_extract(\\\"email_l\\\", '^[^@]+', 0), ''), NULLIF(regexp_extract(\\\"email_r\\\", '^[^@]+', 0), '')) >= 0.88\", \"label_for_charts\": \"Jaro-Winkler >0.88 on username\", \"m_probability\": 0.012500000000000011, \"u_probability\": 0.000501823937001795, \"m_probability_description\": \"Amongst matching record comparisons, 1.25% of records are in the jaro-winkler >0.88 on username comparison level\", \"u_probability_description\": \"Amongst non-matching record comparisons, 0.05% of records are in the jaro-winkler >0.88 on username comparison level\", \"has_tf_adjustments\": false, \"tf_adjustment_column\": null, \"tf_adjustment_weight\": 1.0, \"is_null_level\": false, \"bayes_factor\": 24.909134615384634, \"log2_bayes_factor\": 4.638602995720225, \"comparison_vector_value\": 1, \"max_comparison_vector_value\": 4, \"bayes_factor_description\": \"If comparison level is `jaro-winkler >0.88 on username` then comparison is 24.91 times more likely to be a match\", \"probability_two_random_records_match\": 0.00298012298012298, \"comparison_sort_order\": 4}, {\"comparison_name\": \"email\", \"sql_condition\": \"ELSE\", \"label_for_charts\": \"All other comparisons\", \"m_probability\": 0.012029295425837087, \"u_probability\": 0.9953516949425154, \"m_probability_description\": \"Amongst matching record comparisons, 1.20% of records are in the all other comparisons comparison level\", \"u_probability_description\": \"Amongst non-matching record comparisons, 99.54% of records are in the all other comparisons comparison level\", \"has_tf_adjustments\": false, \"tf_adjustment_column\": null, \"tf_adjustment_weight\": 1.0, \"is_null_level\": false, \"bayes_factor\": 0.012085472388261534, \"log2_bayes_factor\": -6.370582324541905, \"comparison_vector_value\": 0, \"max_comparison_vector_value\": 4, \"bayes_factor_description\": \"If comparison level is `all other comparisons` then comparison is  82.74 times less likely to be a match\", \"probability_two_random_records_match\": 0.00298012298012298, \"comparison_sort_order\": 4}]}}, {\"mode\": \"vega-lite\"});\n",
              "</script>"
            ],
            "text/plain": [
              "alt.VConcatChart(...)"
            ]
          },
          "execution_count": 10,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "linker.visualisations.match_weights_chart()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 11,
      "id": "8576c042",
      "metadata": {
        "execution": {
          "iopub.execute_input": "2024-07-11T13:34:27.001281Z",
          "iopub.status.busy": "2024-07-11T13:34:27.000968Z",
          "iopub.status.idle": "2024-07-11T13:34:27.207585Z",
          "shell.execute_reply": "2024-07-11T13:34:27.206879Z"
        }
      },
      "outputs": [
        {
          "data": {
            "text/html": [
              "\n",
              "<style>\n",
              "  #altair-viz-8dee873cfd504761b05eb53d423af322.vega-embed {\n",
              "    width: 100%;\n",
              "    display: flex;\n",
              "  }\n",
              "\n",
              "  #altair-viz-8dee873cfd504761b05eb53d423af322.vega-embed details,\n",
              "  #altair-viz-8dee873cfd504761b05eb53d423af322.vega-embed details summary {\n",
              "    position: relative;\n",
              "  }\n",
              "</style>\n",
              "<div id=\"altair-viz-8dee873cfd504761b05eb53d423af322\"></div>\n",
              "<script type=\"text/javascript\">\n",
              "  var VEGA_DEBUG = (typeof VEGA_DEBUG == \"undefined\") ? {} : VEGA_DEBUG;\n",
              "  (function(spec, embedOpt){\n",
              "    let outputDiv = document.currentScript.previousElementSibling;\n",
              "    if (outputDiv.id !== \"altair-viz-8dee873cfd504761b05eb53d423af322\") {\n",
              "      outputDiv = document.getElementById(\"altair-viz-8dee873cfd504761b05eb53d423af322\");\n",
              "    }\n",
              "    const paths = {\n",
              "      \"vega\": \"https://cdn.jsdelivr.net/npm/vega@5?noext\",\n",
              "      \"vega-lib\": \"https://cdn.jsdelivr.net/npm/vega-lib?noext\",\n",
              "      \"vega-lite\": \"https://cdn.jsdelivr.net/npm/vega-lite@5.17.0?noext\",\n",
              "      \"vega-embed\": \"https://cdn.jsdelivr.net/npm/vega-embed@6?noext\",\n",
              "    };\n",
              "\n",
              "    function maybeLoadScript(lib, version) {\n",
              "      var key = `${lib.replace(\"-\", \"\")}_version`;\n",
              "      return (VEGA_DEBUG[key] == version) ?\n",
              "        Promise.resolve(paths[lib]) :\n",
              "        new Promise(function(resolve, reject) {\n",
              "          var s = document.createElement('script');\n",
              "          document.getElementsByTagName(\"head\")[0].appendChild(s);\n",
              "          s.async = true;\n",
              "          s.onload = () => {\n",
              "            VEGA_DEBUG[key] = version;\n",
              "            return resolve(paths[lib]);\n",
              "          };\n",
              "          s.onerror = () => reject(`Error loading script: ${paths[lib]}`);\n",
              "          s.src = paths[lib];\n",
              "        });\n",
              "    }\n",
              "\n",
              "    function showError(err) {\n",
              "      outputDiv.innerHTML = `<div class=\"error\" style=\"color:red;\">${err}</div>`;\n",
              "      throw err;\n",
              "    }\n",
              "\n",
              "    function displayChart(vegaEmbed) {\n",
              "      vegaEmbed(outputDiv, spec, embedOpt)\n",
              "        .catch(err => showError(`Javascript Error: ${err.message}<br>This usually means there's a typo in your chart specification. See the javascript console for the full traceback.`));\n",
              "    }\n",
              "\n",
              "    if(typeof define === \"function\" && define.amd) {\n",
              "      requirejs.config({paths});\n",
              "      require([\"vega-embed\"], displayChart, err => showError(`Error loading script: ${err.message}`));\n",
              "    } else {\n",
              "      maybeLoadScript(\"vega\", \"5\")\n",
              "        .then(() => maybeLoadScript(\"vega-lite\", \"5.17.0\"))\n",
              "        .then(() => maybeLoadScript(\"vega-embed\", \"6\"))\n",
              "        .catch(showError)\n",
              "        .then(() => displayChart(vegaEmbed));\n",
              "    }\n",
              "  })({\"config\": {\"view\": {\"continuousWidth\": 300, \"continuousHeight\": 300, \"discreteHeight\": 300, \"discreteWidth\": 400}, \"header\": {\"title\": null}, \"title\": {\"anchor\": \"middle\", \"offset\": 10}}, \"hconcat\": [{\"mark\": \"bar\", \"encoding\": {\"color\": {\"value\": \"green\"}, \"row\": {\"field\": \"comparison_name\", \"header\": {\"labelAlign\": \"left\", \"labelAnchor\": \"middle\", \"labelAngle\": 0}, \"sort\": {\"field\": \"comparison_sort_order\"}, \"type\": \"nominal\"}, \"tooltip\": [{\"field\": \"m_probability_description\", \"title\": \"m probability description\", \"type\": \"nominal\"}, {\"field\": \"comparison_name\", \"title\": \"Comparison column name\", \"type\": \"nominal\"}, {\"field\": \"label_for_charts\", \"title\": \"Label\", \"type\": \"ordinal\"}, {\"field\": \"sql_condition\", \"title\": \"SQL condition\", \"type\": \"nominal\"}, {\"field\": \"m_probability\", \"format\": \".4p\", \"title\": \"m probability\", \"type\": \"quantitative\"}, {\"field\": \"u_probability\", \"format\": \".4p\", \"title\": \"u probability\", \"type\": \"quantitative\"}, {\"field\": \"bayes_factor\", \"format\": \",.4f\", \"title\": \"Bayes factor = m/u\", \"type\": \"quantitative\"}, {\"field\": \"log2_bayes_factor\", \"format\": \",.4f\", \"title\": \"Match weight = log2(m/u)\", \"type\": \"quantitative\"}], \"x\": {\"axis\": {\"title\": \"Proportion of record comparisons\"}, \"field\": \"m_probability\", \"type\": \"quantitative\"}, \"y\": {\"axis\": {\"title\": null}, \"field\": \"label_for_charts\", \"sort\": {\"field\": \"comparison_vector_value\", \"order\": \"descending\"}, \"type\": \"nominal\"}}, \"height\": {\"step\": 12}, \"resolve\": {\"scale\": {\"y\": \"independent\"}}, \"title\": {\"text\": \"Amongst matching record comparisons:\", \"fontSize\": 12, \"fontWeight\": \"bold\"}, \"transform\": [{\"filter\": \"(datum.bayes_factor != 'no-op filter due to vega lite issue 4680')\"}], \"width\": 150}, {\"mark\": \"bar\", \"encoding\": {\"color\": {\"value\": \"red\"}, \"row\": {\"field\": \"comparison_name\", \"header\": {\"labels\": false}, \"sort\": {\"field\": \"comparison_sort_order\"}, \"type\": \"nominal\"}, \"tooltip\": [{\"field\": \"u_probability_description\", \"title\": \"u probability description\", \"type\": \"nominal\"}, {\"field\": \"comparison_name\", \"title\": \"Comparison column name\", \"type\": \"nominal\"}, {\"field\": \"label_for_charts\", \"title\": \"Label\", \"type\": \"ordinal\"}, {\"field\": \"sql_condition\", \"title\": \"SQL condition\", \"type\": \"nominal\"}, {\"field\": \"m_probability\", \"format\": \".4p\", \"title\": \"m probability\", \"type\": \"quantitative\"}, {\"field\": \"u_probability\", \"format\": \".4p\", \"title\": \"u probability\", \"type\": \"quantitative\"}, {\"field\": \"bayes_factor\", \"format\": \",.4f\", \"title\": \"Bayes factor = m/u\", \"type\": \"quantitative\"}, {\"field\": \"log2_bayes_factor\", \"format\": \",.4f\", \"title\": \"Match weight = log2(m/u)\", \"type\": \"quantitative\"}], \"x\": {\"axis\": {\"title\": \"Proportion of record comparisons\"}, \"field\": \"u_probability\", \"type\": \"quantitative\"}, \"y\": {\"axis\": {\"title\": null}, \"field\": \"label_for_charts\", \"sort\": {\"field\": \"comparison_vector_value\", \"order\": \"descending\"}, \"type\": \"nominal\"}}, \"height\": {\"step\": 12}, \"resolve\": {\"scale\": {\"y\": \"independent\"}}, \"title\": {\"text\": \"Amongst non-matching record comparisons:\", \"fontSize\": 12, \"fontWeight\": \"bold\"}, \"transform\": [{\"filter\": \"(datum.bayes_factor != 'no-op filter2 due to vega lite issue 4680')\"}], \"width\": 150}], \"data\": {\"name\": \"data-c0dd00fdb5a295ca2e19be1561fc8105\"}, \"title\": {\"text\": \"Proportion of record comparisons in each comparison level by match status\", \"subtitle\": \"(m and u probabilities)\"}, \"$schema\": \"https://vega.github.io/schema/vega-lite/v5.9.3.json\", \"datasets\": {\"data-c0dd00fdb5a295ca2e19be1561fc8105\": [{\"comparison_name\": \"first_name\", \"sql_condition\": \"\\\"first_name_l\\\" = \\\"first_name_r\\\"\", \"label_for_charts\": \"Exact match on first_name\", \"m_probability\": 0.4914209493176376, \"u_probability\": 0.0057935713975033705, \"m_probability_description\": \"Amongst matching record comparisons, 49.14% of records are in the exact match on first_name comparison level\", \"u_probability_description\": \"Amongst non-matching record comparisons, 0.58% of records are in the exact match on first_name comparison level\", \"has_tf_adjustments\": true, \"tf_adjustment_column\": \"first_name\", \"tf_adjustment_weight\": 1.0, \"is_null_level\": false, \"bayes_factor\": 84.82176460782136, \"log2_bayes_factor\": 6.406362591550452, \"comparison_vector_value\": 4, \"max_comparison_vector_value\": 4, \"bayes_factor_description\": \"If comparison level is `exact match on first_name` then comparison is 84.82 times more likely to be a match\", \"probability_two_random_records_match\": 0.00298012298012298, \"comparison_sort_order\": 0}, {\"comparison_name\": \"first_name\", \"sql_condition\": \"jaro_winkler_similarity(\\\"first_name_l\\\", \\\"first_name_r\\\") >= 0.92\", \"label_for_charts\": \"Jaro-Winkler distance of first_name >= 0.92\", \"m_probability\": 0.15176057384758346, \"u_probability\": 0.0023429457903817435, \"m_probability_description\": \"Amongst matching record comparisons, 15.18% of records are in the jaro-winkler distance of first_name >= 0.92 comparison level\", \"u_probability_description\": \"Amongst non-matching record comparisons, 0.23% of records are in the jaro-winkler distance of first_name >= 0.92 comparison level\", \"has_tf_adjustments\": false, \"tf_adjustment_column\": null, \"tf_adjustment_weight\": 1.0, \"is_null_level\": false, \"bayes_factor\": 64.77340383656791, \"log2_bayes_factor\": 6.017329654456136, \"comparison_vector_value\": 3, \"max_comparison_vector_value\": 4, \"bayes_factor_description\": \"If comparison level is `jaro-winkler distance of first_name >= 0.92` then comparison is 64.77 times more likely to be a match\", \"probability_two_random_records_match\": 0.00298012298012298, \"comparison_sort_order\": 0}, {\"comparison_name\": \"first_name\", \"sql_condition\": \"jaro_winkler_similarity(\\\"first_name_l\\\", \\\"first_name_r\\\") >= 0.88\", \"label_for_charts\": \"Jaro-Winkler distance of first_name >= 0.88\", \"m_probability\": 0.07406496776118927, \"u_probability\": 0.0015484319951285285, \"m_probability_description\": \"Amongst matching record comparisons, 7.41% of records are in the jaro-winkler distance of first_name >= 0.88 comparison level\", \"u_probability_description\": \"Amongst non-matching record comparisons, 0.15% of records are in the jaro-winkler distance of first_name >= 0.88 comparison level\", \"has_tf_adjustments\": false, \"tf_adjustment_column\": null, \"tf_adjustment_weight\": 1.0, \"is_null_level\": false, \"bayes_factor\": 47.83223802801974, \"log2_bayes_factor\": 5.579911390213005, \"comparison_vector_value\": 2, \"max_comparison_vector_value\": 4, \"bayes_factor_description\": \"If comparison level is `jaro-winkler distance of first_name >= 0.88` then comparison is 47.83 times more likely to be a match\", \"probability_two_random_records_match\": 0.00298012298012298, \"comparison_sort_order\": 0}, {\"comparison_name\": \"first_name\", \"sql_condition\": \"jaro_winkler_similarity(\\\"first_name_l\\\", \\\"first_name_r\\\") >= 0.7\", \"label_for_charts\": \"Jaro-Winkler distance of first_name >= 0.7\", \"m_probability\": 0.0790861077150486, \"u_probability\": 0.018934945558406913, \"m_probability_description\": \"Amongst matching record comparisons, 7.91% of records are in the jaro-winkler distance of first_name >= 0.7 comparison level\", \"u_probability_description\": \"Amongst non-matching record comparisons, 1.89% of records are in the jaro-winkler distance of first_name >= 0.7 comparison level\", \"has_tf_adjustments\": false, \"tf_adjustment_column\": null, \"tf_adjustment_weight\": 1.0, \"is_null_level\": false, \"bayes_factor\": 4.176727494203711, \"log2_bayes_factor\": 2.0623730195874392, \"comparison_vector_value\": 1, \"max_comparison_vector_value\": 4, \"bayes_factor_description\": \"If comparison level is `jaro-winkler distance of first_name >= 0.7` then comparison is 4.18 times more likely to be a match\", \"probability_two_random_records_match\": 0.00298012298012298, \"comparison_sort_order\": 0}, {\"comparison_name\": \"first_name\", \"sql_condition\": \"ELSE\", \"label_for_charts\": \"All other comparisons\", \"m_probability\": 0.2036674013585411, \"u_probability\": 0.9713801052585794, \"m_probability_description\": \"Amongst matching record comparisons, 20.37% of records are in the all other comparisons comparison level\", \"u_probability_description\": \"Amongst non-matching record comparisons, 97.14% of records are in the all other comparisons comparison level\", \"has_tf_adjustments\": false, \"tf_adjustment_column\": null, \"tf_adjustment_weight\": 1.0, \"is_null_level\": false, \"bayes_factor\": 0.2096680797310804, \"log2_bayes_factor\": -2.25382085538399, \"comparison_vector_value\": 0, \"max_comparison_vector_value\": 4, \"bayes_factor_description\": \"If comparison level is `all other comparisons` then comparison is  4.77 times less likely to be a match\", \"probability_two_random_records_match\": 0.00298012298012298, \"comparison_sort_order\": 0}, {\"comparison_name\": \"surname\", \"sql_condition\": \"\\\"surname_l\\\" = \\\"surname_r\\\"\", \"label_for_charts\": \"Exact match on surname\", \"m_probability\": 0.43457460622893695, \"u_probability\": 0.004889975550122249, \"m_probability_description\": \"Amongst matching record comparisons, 43.46% of records are in the exact match on surname comparison level\", \"u_probability_description\": \"Amongst non-matching record comparisons, 0.49% of records are in the exact match on surname comparison level\", \"has_tf_adjustments\": true, \"tf_adjustment_column\": \"surname\", \"tf_adjustment_weight\": 1.0, \"is_null_level\": false, \"bayes_factor\": 88.87050697381761, \"log2_bayes_factor\": 6.47363281313057, \"comparison_vector_value\": 4, \"max_comparison_vector_value\": 4, \"bayes_factor_description\": \"If comparison level is `exact match on surname` then comparison is 88.87 times more likely to be a match\", \"probability_two_random_records_match\": 0.00298012298012298, \"comparison_sort_order\": 1}, {\"comparison_name\": \"surname\", \"sql_condition\": \"jaro_winkler_similarity(\\\"surname_l\\\", \\\"surname_r\\\") >= 0.92\", \"label_for_charts\": \"Jaro-Winkler distance of surname >= 0.92\", \"m_probability\": 0.22529103510053114, \"u_probability\": 0.00283905173880724, \"m_probability_description\": \"Amongst matching record comparisons, 22.53% of records are in the jaro-winkler distance of surname >= 0.92 comparison level\", \"u_probability_description\": \"Amongst non-matching record comparisons, 0.28% of records are in the jaro-winkler distance of surname >= 0.92 comparison level\", \"has_tf_adjustments\": false, \"tf_adjustment_column\": null, \"tf_adjustment_weight\": 1.0, \"is_null_level\": false, \"bayes_factor\": 79.35432525621452, \"log2_bayes_factor\": 6.3102369550698105, \"comparison_vector_value\": 3, \"max_comparison_vector_value\": 4, \"bayes_factor_description\": \"If comparison level is `jaro-winkler distance of surname >= 0.92` then comparison is 79.35 times more likely to be a match\", \"probability_two_random_records_match\": 0.00298012298012298, \"comparison_sort_order\": 1}, {\"comparison_name\": \"surname\", \"sql_condition\": \"jaro_winkler_similarity(\\\"surname_l\\\", \\\"surname_r\\\") >= 0.88\", \"label_for_charts\": \"Jaro-Winkler distance of surname >= 0.88\", \"m_probability\": 0.04432221456939868, \"u_probability\": 0.0011314412292407403, \"m_probability_description\": \"Amongst matching record comparisons, 4.43% of records are in the jaro-winkler distance of surname >= 0.88 comparison level\", \"u_probability_description\": \"Amongst non-matching record comparisons, 0.11% of records are in the jaro-winkler distance of surname >= 0.88 comparison level\", \"has_tf_adjustments\": false, \"tf_adjustment_column\": null, \"tf_adjustment_weight\": 1.0, \"is_null_level\": false, \"bayes_factor\": 39.17323624413204, \"log2_bayes_factor\": 5.291796414406481, \"comparison_vector_value\": 2, \"max_comparison_vector_value\": 4, \"bayes_factor_description\": \"If comparison level is `jaro-winkler distance of surname >= 0.88` then comparison is 39.17 times more likely to be a match\", \"probability_two_random_records_match\": 0.00298012298012298, \"comparison_sort_order\": 1}, {\"comparison_name\": \"surname\", \"sql_condition\": \"jaro_winkler_similarity(\\\"surname_l\\\", \\\"surname_r\\\") >= 0.7\", \"label_for_charts\": \"Jaro-Winkler distance of surname >= 0.7\", \"m_probability\": 0.07784087870954867, \"u_probability\": 0.014729633311540402, \"m_probability_description\": \"Amongst matching record comparisons, 7.78% of records are in the jaro-winkler distance of surname >= 0.7 comparison level\", \"u_probability_description\": \"Amongst non-matching record comparisons, 1.47% of records are in the jaro-winkler distance of surname >= 0.7 comparison level\", \"has_tf_adjustments\": false, \"tf_adjustment_column\": null, \"tf_adjustment_weight\": 1.0, \"is_null_level\": false, \"bayes_factor\": 5.28464470656997, \"log2_bayes_factor\": 2.4018064806335238, \"comparison_vector_value\": 1, \"max_comparison_vector_value\": 4, \"bayes_factor_description\": \"If comparison level is `jaro-winkler distance of surname >= 0.7` then comparison is 5.28 times more likely to be a match\", \"probability_two_random_records_match\": 0.00298012298012298, \"comparison_sort_order\": 1}, {\"comparison_name\": \"surname\", \"sql_condition\": \"ELSE\", \"label_for_charts\": \"All other comparisons\", \"m_probability\": 0.2179712653915845, \"u_probability\": 0.9764098981702893, \"m_probability_description\": \"Amongst matching record comparisons, 21.80% of records are in the all other comparisons comparison level\", \"u_probability_description\": \"Amongst non-matching record comparisons, 97.64% of records are in the all other comparisons comparison level\", \"has_tf_adjustments\": false, \"tf_adjustment_column\": null, \"tf_adjustment_weight\": 1.0, \"is_null_level\": false, \"bayes_factor\": 0.22323745980048387, \"log2_bayes_factor\": -2.1633489595936646, \"comparison_vector_value\": 0, \"max_comparison_vector_value\": 4, \"bayes_factor_description\": \"If comparison level is `all other comparisons` then comparison is  4.48 times less likely to be a match\", \"probability_two_random_records_match\": 0.00298012298012298, \"comparison_sort_order\": 1}, {\"comparison_name\": \"dob\", \"sql_condition\": \"\\\"dob_l\\\" = \\\"dob_r\\\"\", \"label_for_charts\": \"Exact match on dob\", \"m_probability\": 0.3914216652882992, \"u_probability\": 0.0017477477477477479, \"m_probability_description\": \"Amongst matching record comparisons, 39.14% of records are in the exact match on dob comparison level\", \"u_probability_description\": \"Amongst non-matching record comparisons, 0.17% of records are in the exact match on dob comparison level\", \"has_tf_adjustments\": false, \"tf_adjustment_column\": null, \"tf_adjustment_weight\": 1.0, \"is_null_level\": false, \"bayes_factor\": 223.95775694330518, \"log2_bayes_factor\": 7.807082825648326, \"comparison_vector_value\": 2, \"max_comparison_vector_value\": 2, \"bayes_factor_description\": \"If comparison level is `exact match on dob` then comparison is 223.96 times more likely to be a match\", \"probability_two_random_records_match\": 0.00298012298012298, \"comparison_sort_order\": 2}, {\"comparison_name\": \"dob\", \"sql_condition\": \"levenshtein(\\\"dob_l\\\", \\\"dob_r\\\") <= 1\", \"label_for_charts\": \"Levenshtein distance of dob <= 1\", \"m_probability\": 0.14937817941895074, \"u_probability\": 0.0016016016016016017, \"m_probability_description\": \"Amongst matching record comparisons, 14.94% of records are in the levenshtein distance of dob <= 1 comparison level\", \"u_probability_description\": \"Amongst non-matching record comparisons, 0.16% of records are in the levenshtein distance of dob <= 1 comparison level\", \"has_tf_adjustments\": false, \"tf_adjustment_column\": null, \"tf_adjustment_weight\": 1.0, \"is_null_level\": false, \"bayes_factor\": 93.26800077470736, \"log2_bayes_factor\": 6.543310288053254, \"comparison_vector_value\": 1, \"max_comparison_vector_value\": 2, \"bayes_factor_description\": \"If comparison level is `levenshtein distance of dob <= 1` then comparison is 93.27 times more likely to be a match\", \"probability_two_random_records_match\": 0.00298012298012298, \"comparison_sort_order\": 2}, {\"comparison_name\": \"dob\", \"sql_condition\": \"ELSE\", \"label_for_charts\": \"All other comparisons\", \"m_probability\": 0.45920015529275016, \"u_probability\": 0.9966506506506506, \"m_probability_description\": \"Amongst matching record comparisons, 45.92% of records are in the all other comparisons comparison level\", \"u_probability_description\": \"Amongst non-matching record comparisons, 99.67% of records are in the all other comparisons comparison level\", \"has_tf_adjustments\": false, \"tf_adjustment_column\": null, \"tf_adjustment_weight\": 1.0, \"is_null_level\": false, \"bayes_factor\": 0.4607433457179476, \"log2_bayes_factor\": -1.1179647649923983, \"comparison_vector_value\": 0, \"max_comparison_vector_value\": 2, \"bayes_factor_description\": \"If comparison level is `all other comparisons` then comparison is  2.17 times less likely to be a match\", \"probability_two_random_records_match\": 0.00298012298012298, \"comparison_sort_order\": 2}, {\"comparison_name\": \"city\", \"sql_condition\": \"\\\"city_l\\\" = \\\"city_r\\\"\", \"label_for_charts\": \"Exact match on city\", \"m_probability\": 0.562574722357491, \"u_probability\": 0.0551475711801453, \"m_probability_description\": \"Amongst matching record comparisons, 56.26% of records are in the exact match on city comparison level\", \"u_probability_description\": \"Amongst non-matching record comparisons, 5.51% of records are in the exact match on city comparison level\", \"has_tf_adjustments\": true, \"tf_adjustment_column\": \"city\", \"tf_adjustment_weight\": 1.0, \"is_null_level\": false, \"bayes_factor\": 10.201260188228089, \"log2_bayes_factor\": 3.350675477967205, \"comparison_vector_value\": 1, \"max_comparison_vector_value\": 1, \"bayes_factor_description\": \"If comparison level is `exact match on city` then comparison is 10.20 times more likely to be a match\", \"probability_two_random_records_match\": 0.00298012298012298, \"comparison_sort_order\": 3}, {\"comparison_name\": \"city\", \"sql_condition\": \"ELSE\", \"label_for_charts\": \"All other comparisons\", \"m_probability\": 0.4374252776425091, \"u_probability\": 0.9448524288198547, \"m_probability_description\": \"Amongst matching record comparisons, 43.74% of records are in the all other comparisons comparison level\", \"u_probability_description\": \"Amongst non-matching record comparisons, 94.49% of records are in the all other comparisons comparison level\", \"has_tf_adjustments\": false, \"tf_adjustment_column\": null, \"tf_adjustment_weight\": 1.0, \"is_null_level\": false, \"bayes_factor\": 0.46295618691361645, \"log2_bayes_factor\": -1.111052428212497, \"comparison_vector_value\": 0, \"max_comparison_vector_value\": 1, \"bayes_factor_description\": \"If comparison level is `all other comparisons` then comparison is  2.16 times less likely to be a match\", \"probability_two_random_records_match\": 0.00298012298012298, \"comparison_sort_order\": 3}, {\"comparison_name\": \"email\", \"sql_condition\": \"\\\"email_l\\\" = \\\"email_r\\\"\", \"label_for_charts\": \"Exact match on email\", \"m_probability\": 0.5529665836585227, \"u_probability\": 0.0021938713143283602, \"m_probability_description\": \"Amongst matching record comparisons, 55.30% of records are in the exact match on email comparison level\", \"u_probability_description\": \"Amongst non-matching record comparisons, 0.22% of records are in the exact match on email comparison level\", \"has_tf_adjustments\": true, \"tf_adjustment_column\": \"email\", \"tf_adjustment_weight\": 1.0, \"is_null_level\": false, \"bayes_factor\": 252.0506011665547, \"log2_bayes_factor\": 7.977569585101858, \"comparison_vector_value\": 4, \"max_comparison_vector_value\": 4, \"bayes_factor_description\": \"If comparison level is `exact match on email` then comparison is 252.05 times more likely to be a match\", \"probability_two_random_records_match\": 0.00298012298012298, \"comparison_sort_order\": 4}, {\"comparison_name\": \"email\", \"sql_condition\": \"NULLIF(regexp_extract(\\\"email_l\\\", '^[^@]+', 0), '') = NULLIF(regexp_extract(\\\"email_r\\\", '^[^@]+', 0), '')\", \"label_for_charts\": \"Exact match on username\", \"m_probability\": 0.2208741262673713, \"u_probability\": 0.0010390328952024346, \"m_probability_description\": \"Amongst matching record comparisons, 22.09% of records are in the exact match on username comparison level\", \"u_probability_description\": \"Amongst non-matching record comparisons, 0.10% of records are in the exact match on username comparison level\", \"has_tf_adjustments\": false, \"tf_adjustment_column\": null, \"tf_adjustment_weight\": 1.0, \"is_null_level\": false, \"bayes_factor\": 212.57664438462118, \"log2_bayes_factor\": 7.731839287671784, \"comparison_vector_value\": 3, \"max_comparison_vector_value\": 4, \"bayes_factor_description\": \"If comparison level is `exact match on username` then comparison is 212.58 times more likely to be a match\", \"probability_two_random_records_match\": 0.00298012298012298, \"comparison_sort_order\": 4}, {\"comparison_name\": \"email\", \"sql_condition\": \"jaro_winkler_similarity(\\\"email_l\\\", \\\"email_r\\\") >= 0.88\", \"label_for_charts\": \"Jaro-Winkler distance of email >= 0.88\", \"m_probability\": 0.2141299946482688, \"u_probability\": 0.0009135769109519858, \"m_probability_description\": \"Amongst matching record comparisons, 21.41% of records are in the jaro-winkler distance of email >= 0.88 comparison level\", \"u_probability_description\": \"Amongst non-matching record comparisons, 0.09% of records are in the jaro-winkler distance of email >= 0.88 comparison level\", \"has_tf_adjustments\": false, \"tf_adjustment_column\": null, \"tf_adjustment_weight\": 1.0, \"is_null_level\": false, \"bayes_factor\": 234.38639055045326, \"log2_bayes_factor\": 7.872744993087004, \"comparison_vector_value\": 2, \"max_comparison_vector_value\": 4, \"bayes_factor_description\": \"If comparison level is `jaro-winkler distance of email >= 0.88` then comparison is 234.39 times more likely to be a match\", \"probability_two_random_records_match\": 0.00298012298012298, \"comparison_sort_order\": 4}, {\"comparison_name\": \"email\", \"sql_condition\": \"jaro_winkler_similarity(NULLIF(regexp_extract(\\\"email_l\\\", '^[^@]+', 0), ''), NULLIF(regexp_extract(\\\"email_r\\\", '^[^@]+', 0), '')) >= 0.88\", \"label_for_charts\": \"Jaro-Winkler >0.88 on username\", \"m_probability\": 0.012500000000000011, \"u_probability\": 0.000501823937001795, \"m_probability_description\": \"Amongst matching record comparisons, 1.25% of records are in the jaro-winkler >0.88 on username comparison level\", \"u_probability_description\": \"Amongst non-matching record comparisons, 0.05% of records are in the jaro-winkler >0.88 on username comparison level\", \"has_tf_adjustments\": false, \"tf_adjustment_column\": null, \"tf_adjustment_weight\": 1.0, \"is_null_level\": false, \"bayes_factor\": 24.909134615384634, \"log2_bayes_factor\": 4.638602995720225, \"comparison_vector_value\": 1, \"max_comparison_vector_value\": 4, \"bayes_factor_description\": \"If comparison level is `jaro-winkler >0.88 on username` then comparison is 24.91 times more likely to be a match\", \"probability_two_random_records_match\": 0.00298012298012298, \"comparison_sort_order\": 4}, {\"comparison_name\": \"email\", \"sql_condition\": \"ELSE\", \"label_for_charts\": \"All other comparisons\", \"m_probability\": 0.012029295425837087, \"u_probability\": 0.9953516949425154, \"m_probability_description\": \"Amongst matching record comparisons, 1.20% of records are in the all other comparisons comparison level\", \"u_probability_description\": \"Amongst non-matching record comparisons, 99.54% of records are in the all other comparisons comparison level\", \"has_tf_adjustments\": false, \"tf_adjustment_column\": null, \"tf_adjustment_weight\": 1.0, \"is_null_level\": false, \"bayes_factor\": 0.012085472388261534, \"log2_bayes_factor\": -6.370582324541905, \"comparison_vector_value\": 0, \"max_comparison_vector_value\": 4, \"bayes_factor_description\": \"If comparison level is `all other comparisons` then comparison is  82.74 times less likely to be a match\", \"probability_two_random_records_match\": 0.00298012298012298, \"comparison_sort_order\": 4}]}}, {\"mode\": \"vega-lite\"});\n",
              "</script>"
            ],
            "text/plain": [
              "alt.HConcatChart(...)"
            ]
          },
          "execution_count": 11,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "linker.visualisations.m_u_parameters_chart()"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "1f58a85a",
      "metadata": {},
      "source": [
        "We can also compare the estimates that were produced by the different EM training sessions"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 12,
      "id": "e267dc50",
      "metadata": {
        "execution": {
          "iopub.execute_input": "2024-07-11T13:34:27.211362Z",
          "iopub.status.busy": "2024-07-11T13:34:27.211072Z",
          "iopub.status.idle": "2024-07-11T13:34:27.378196Z",
          "shell.execute_reply": "2024-07-11T13:34:27.377546Z"
        }
      },
      "outputs": [
        {
          "data": {
            "text/html": [
              "\n",
              "<style>\n",
              "  #altair-viz-f63604f7f1374522b1f7813f0bc868d4.vega-embed {\n",
              "    width: 100%;\n",
              "    display: flex;\n",
              "  }\n",
              "\n",
              "  #altair-viz-f63604f7f1374522b1f7813f0bc868d4.vega-embed details,\n",
              "  #altair-viz-f63604f7f1374522b1f7813f0bc868d4.vega-embed details summary {\n",
              "    position: relative;\n",
              "  }\n",
              "</style>\n",
              "<div id=\"altair-viz-f63604f7f1374522b1f7813f0bc868d4\"></div>\n",
              "<script type=\"text/javascript\">\n",
              "  var VEGA_DEBUG = (typeof VEGA_DEBUG == \"undefined\") ? {} : VEGA_DEBUG;\n",
              "  (function(spec, embedOpt){\n",
              "    let outputDiv = document.currentScript.previousElementSibling;\n",
              "    if (outputDiv.id !== \"altair-viz-f63604f7f1374522b1f7813f0bc868d4\") {\n",
              "      outputDiv = document.getElementById(\"altair-viz-f63604f7f1374522b1f7813f0bc868d4\");\n",
              "    }\n",
              "    const paths = {\n",
              "      \"vega\": \"https://cdn.jsdelivr.net/npm/vega@5?noext\",\n",
              "      \"vega-lib\": \"https://cdn.jsdelivr.net/npm/vega-lib?noext\",\n",
              "      \"vega-lite\": \"https://cdn.jsdelivr.net/npm/vega-lite@5.17.0?noext\",\n",
              "      \"vega-embed\": \"https://cdn.jsdelivr.net/npm/vega-embed@6?noext\",\n",
              "    };\n",
              "\n",
              "    function maybeLoadScript(lib, version) {\n",
              "      var key = `${lib.replace(\"-\", \"\")}_version`;\n",
              "      return (VEGA_DEBUG[key] == version) ?\n",
              "        Promise.resolve(paths[lib]) :\n",
              "        new Promise(function(resolve, reject) {\n",
              "          var s = document.createElement('script');\n",
              "          document.getElementsByTagName(\"head\")[0].appendChild(s);\n",
              "          s.async = true;\n",
              "          s.onload = () => {\n",
              "            VEGA_DEBUG[key] = version;\n",
              "            return resolve(paths[lib]);\n",
              "          };\n",
              "          s.onerror = () => reject(`Error loading script: ${paths[lib]}`);\n",
              "          s.src = paths[lib];\n",
              "        });\n",
              "    }\n",
              "\n",
              "    function showError(err) {\n",
              "      outputDiv.innerHTML = `<div class=\"error\" style=\"color:red;\">${err}</div>`;\n",
              "      throw err;\n",
              "    }\n",
              "\n",
              "    function displayChart(vegaEmbed) {\n",
              "      vegaEmbed(outputDiv, spec, embedOpt)\n",
              "        .catch(err => showError(`Javascript Error: ${err.message}<br>This usually means there's a typo in your chart specification. See the javascript console for the full traceback.`));\n",
              "    }\n",
              "\n",
              "    if(typeof define === \"function\" && define.amd) {\n",
              "      requirejs.config({paths});\n",
              "      require([\"vega-embed\"], displayChart, err => showError(`Error loading script: ${err.message}`));\n",
              "    } else {\n",
              "      maybeLoadScript(\"vega\", \"5\")\n",
              "        .then(() => maybeLoadScript(\"vega-lite\", \"5.17.0\"))\n",
              "        .then(() => maybeLoadScript(\"vega-embed\", \"6\"))\n",
              "        .catch(showError)\n",
              "        .then(() => displayChart(vegaEmbed));\n",
              "    }\n",
              "  })({\"config\": {\"view\": {\"continuousWidth\": 400, \"continuousHeight\": 300}, \"title\": {\"anchor\": \"middle\", \"fontSize\": 18, \"subtitleFontSize\": 14}}, \"data\": {\"name\": \"data-eed229f9af66f64c5d674539ab22bf22\"}, \"mark\": {\"type\": \"point\", \"filled\": false, \"opacity\": 0.7, \"size\": 100}, \"encoding\": {\"color\": {\"field\": \"estimate_description\", \"type\": \"nominal\"}, \"column\": {\"align\": \"each\", \"field\": \"col_header\", \"header\": {\"labelFontSize\": 14, \"labelFontWeight\": \"bold\"}, \"title\": null, \"type\": \"nominal\"}, \"row\": {\"align\": \"each\", \"field\": \"comparison_name\", \"header\": {\"labelAlign\": \"left\", \"labelAnchor\": \"middle\", \"labelAngle\": 0, \"labelFontSize\": 12, \"labelFontWeight\": \"bold\"}, \"sort\": {\"field\": \"comparison_sort_order\"}, \"title\": null, \"type\": \"nominal\"}, \"shape\": {\"field\": \"estimate_description\", \"scale\": {\"range\": [\"circle\", \"square\", \"triangle\", \"diamond\"]}, \"type\": \"nominal\"}, \"tooltip\": [{\"field\": \"comparison_name\", \"type\": \"nominal\"}, {\"field\": \"estimate_description\", \"type\": \"nominal\"}, {\"field\": \"estimated_probability\", \"type\": \"quantitative\"}], \"x\": {\"axis\": {\"gridColor\": {\"condition\": {\"test\": \"abs(datum.value / 10)  <= 1 & datum.value % 10 === 0\", \"value\": \"#aaa\"}, \"value\": \"#ddd\"}, \"gridDash\": {\"condition\": {\"test\": \"abs(datum.value / 10) == 1\", \"value\": [3]}, \"value\": null}, \"gridWidth\": {\"condition\": {\"test\": \"abs(datum.value / 10)  <= 1 & datum.value % 10 === 0\", \"value\": 2}, \"value\": 1}}, \"field\": \"estimated_probability_as_log_odds\", \"title\": \"Estimated probability as log odds\", \"type\": \"quantitative\"}, \"y\": {\"axis\": {\"grid\": true, \"title\": null}, \"field\": \"comparison_level_label\", \"sort\": {\"field\": \"comparison_vector_value\", \"order\": \"descending\"}, \"type\": \"nominal\"}}, \"params\": [{\"name\": \"mouse_zoom\", \"select\": {\"type\": \"interval\", \"encodings\": [\"x\"]}, \"bind\": \"scales\"}], \"resolve\": {\"scale\": {\"y\": \"independent\"}}, \"title\": {\"text\": \"Comparison of parameter estimates across training sessions\", \"subtitle\": \"Use mousewheel to zoom\"}, \"transform\": [{\"calculate\": \"datum.m_or_u + '-probability (as log odds)'\", \"as\": \"col_header\"}], \"$schema\": \"https://vega.github.io/schema/vega-lite/v5.9.3.json\", \"datasets\": {\"data-eed229f9af66f64c5d674539ab22bf22\": [{\"m_or_u\": \"m\", \"estimated_probability\": 0.4914209493176376, \"estimate_description\": \"EM, blocked on: l.\\\"dob\\\" = r.\\\"dob\\\"\", \"estimated_probability_as_log_odds\": -0.04951267473254497, \"sql_condition\": \"\\\"first_name_l\\\" = \\\"first_name_r\\\"\", \"comparison_level_label\": \"Exact match on first_name\", \"comparison_vector_value\": 4, \"comparison_name\": \"first_name\", \"comparison_sort_order\": 0}, {\"m_or_u\": \"m\", \"estimated_probability\": 0.15176057384758346, \"estimate_description\": \"EM, blocked on: l.\\\"dob\\\" = r.\\\"dob\\\"\", \"estimated_probability_as_log_odds\": -2.4826745018402243, \"sql_condition\": \"jaro_winkler_similarity(\\\"first_name_l\\\", \\\"first_name_r\\\") >= 0.92\", \"comparison_level_label\": \"Jaro-Winkler distance of first_name >= 0.92\", \"comparison_vector_value\": 3, \"comparison_name\": \"first_name\", \"comparison_sort_order\": 0}, {\"m_or_u\": \"m\", \"estimated_probability\": 0.07406496776118927, \"estimate_description\": \"EM, blocked on: l.\\\"dob\\\" = r.\\\"dob\\\"\", \"estimated_probability_as_log_odds\": -3.644047747288593, \"sql_condition\": \"jaro_winkler_similarity(\\\"first_name_l\\\", \\\"first_name_r\\\") >= 0.88\", \"comparison_level_label\": \"Jaro-Winkler distance of first_name >= 0.88\", \"comparison_vector_value\": 2, \"comparison_name\": \"first_name\", \"comparison_sort_order\": 0}, {\"m_or_u\": \"m\", \"estimated_probability\": 0.0790861077150486, \"estimate_description\": \"EM, blocked on: l.\\\"dob\\\" = r.\\\"dob\\\"\", \"estimated_probability_as_log_odds\": -3.5415700691823817, \"sql_condition\": \"jaro_winkler_similarity(\\\"first_name_l\\\", \\\"first_name_r\\\") >= 0.7\", \"comparison_level_label\": \"Jaro-Winkler distance of first_name >= 0.7\", \"comparison_vector_value\": 1, \"comparison_name\": \"first_name\", \"comparison_sort_order\": 0}, {\"m_or_u\": \"m\", \"estimated_probability\": 0.2036674013585411, \"estimate_description\": \"EM, blocked on: l.\\\"dob\\\" = r.\\\"dob\\\"\", \"estimated_probability_as_log_odds\": -1.9671560333554556, \"sql_condition\": \"ELSE\", \"comparison_level_label\": \"All other comparisons\", \"comparison_vector_value\": 0, \"comparison_name\": \"first_name\", \"comparison_sort_order\": 0}, {\"m_or_u\": \"m\", \"estimated_probability\": 0.43457460622893695, \"estimate_description\": \"EM, blocked on: l.\\\"dob\\\" = r.\\\"dob\\\"\", \"estimated_probability_as_log_odds\": -0.37973280230037454, \"sql_condition\": \"\\\"surname_l\\\" = \\\"surname_r\\\"\", \"comparison_level_label\": \"Exact match on surname\", \"comparison_vector_value\": 4, \"comparison_name\": \"surname\", \"comparison_sort_order\": 1}, {\"m_or_u\": \"m\", \"estimated_probability\": 0.22529103510053114, \"estimate_description\": \"EM, blocked on: l.\\\"dob\\\" = r.\\\"dob\\\"\", \"estimated_probability_as_log_odds\": -1.781864528340751, \"sql_condition\": \"jaro_winkler_similarity(\\\"surname_l\\\", \\\"surname_r\\\") >= 0.92\", \"comparison_level_label\": \"Jaro-Winkler distance of surname >= 0.92\", \"comparison_vector_value\": 3, \"comparison_name\": \"surname\", \"comparison_sort_order\": 1}, {\"m_or_u\": \"m\", \"estimated_probability\": 0.04432221456939868, \"estimate_description\": \"EM, blocked on: l.\\\"dob\\\" = r.\\\"dob\\\"\", \"estimated_probability_as_log_odds\": -4.430422410891006, \"sql_condition\": \"jaro_winkler_similarity(\\\"surname_l\\\", \\\"surname_r\\\") >= 0.88\", \"comparison_level_label\": \"Jaro-Winkler distance of surname >= 0.88\", \"comparison_vector_value\": 2, \"comparison_name\": \"surname\", \"comparison_sort_order\": 1}, {\"m_or_u\": \"m\", \"estimated_probability\": 0.07784087870954867, \"estimate_description\": \"EM, blocked on: l.\\\"dob\\\" = r.\\\"dob\\\"\", \"estimated_probability_as_log_odds\": -3.566415812203307, \"sql_condition\": \"jaro_winkler_similarity(\\\"surname_l\\\", \\\"surname_r\\\") >= 0.7\", \"comparison_level_label\": \"Jaro-Winkler distance of surname >= 0.7\", \"comparison_vector_value\": 1, \"comparison_name\": \"surname\", \"comparison_sort_order\": 1}, {\"m_or_u\": \"m\", \"estimated_probability\": 0.2179712653915845, \"estimate_description\": \"EM, blocked on: l.\\\"dob\\\" = r.\\\"dob\\\"\", \"estimated_probability_as_log_odds\": -1.8430836577748586, \"sql_condition\": \"ELSE\", \"comparison_level_label\": \"All other comparisons\", \"comparison_vector_value\": 0, \"comparison_name\": \"surname\", \"comparison_sort_order\": 1}, {\"m_or_u\": \"m\", \"estimated_probability\": 0.3914216652882992, \"estimate_description\": \"EM, blocked on: (l.\\\"first_name\\\" = r.\\\"first_name\\\") AND (l.\\\"surname\\\" = r.\\\"surname\\\")\", \"estimated_probability_as_log_odds\": -0.636719363350654, \"sql_condition\": \"\\\"dob_l\\\" = \\\"dob_r\\\"\", \"comparison_level_label\": \"Exact match on dob\", \"comparison_vector_value\": 2, \"comparison_name\": \"dob\", \"comparison_sort_order\": 2}, {\"m_or_u\": \"m\", \"estimated_probability\": 0.14937817941895074, \"estimate_description\": \"EM, blocked on: (l.\\\"first_name\\\" = r.\\\"first_name\\\") AND (l.\\\"surname\\\" = r.\\\"surname\\\")\", \"estimated_probability_as_log_odds\": -2.5095484439202043, \"sql_condition\": \"levenshtein(\\\"dob_l\\\", \\\"dob_r\\\") <= 1\", \"comparison_level_label\": \"Levenshtein distance of dob <= 1\", \"comparison_vector_value\": 1, \"comparison_name\": \"dob\", \"comparison_sort_order\": 2}, {\"m_or_u\": \"m\", \"estimated_probability\": 0.45920015529275016, \"estimate_description\": \"EM, blocked on: (l.\\\"first_name\\\" = r.\\\"first_name\\\") AND (l.\\\"surname\\\" = r.\\\"surname\\\")\", \"estimated_probability_as_log_odds\": -0.23597160743906118, \"sql_condition\": \"ELSE\", \"comparison_level_label\": \"All other comparisons\", \"comparison_vector_value\": 0, \"comparison_name\": \"dob\", \"comparison_sort_order\": 2}, {\"m_or_u\": \"m\", \"estimated_probability\": 0.5688095702551735, \"estimate_description\": \"EM, blocked on: (l.\\\"first_name\\\" = r.\\\"first_name\\\") AND (l.\\\"surname\\\" = r.\\\"surname\\\")\", \"estimated_probability_as_log_odds\": 0.3996205807079716, \"sql_condition\": \"\\\"city_l\\\" = \\\"city_r\\\"\", \"comparison_level_label\": \"Exact match on city\", \"comparison_vector_value\": 1, \"comparison_name\": \"city\", \"comparison_sort_order\": 3}, {\"m_or_u\": \"m\", \"estimated_probability\": 0.5563398744598084, \"estimate_description\": \"EM, blocked on: l.\\\"dob\\\" = r.\\\"dob\\\"\", \"estimated_probability_as_log_odds\": 0.3265116162283546, \"sql_condition\": \"\\\"city_l\\\" = \\\"city_r\\\"\", \"comparison_level_label\": \"Exact match on city\", \"comparison_vector_value\": 1, \"comparison_name\": \"city\", \"comparison_sort_order\": 3}, {\"m_or_u\": \"m\", \"estimated_probability\": 0.4311904297448265, \"estimate_description\": \"EM, blocked on: (l.\\\"first_name\\\" = r.\\\"first_name\\\") AND (l.\\\"surname\\\" = r.\\\"surname\\\")\", \"estimated_probability_as_log_odds\": -0.3996205807079711, \"sql_condition\": \"ELSE\", \"comparison_level_label\": \"All other comparisons\", \"comparison_vector_value\": 0, \"comparison_name\": \"city\", \"comparison_sort_order\": 3}, {\"m_or_u\": \"m\", \"estimated_probability\": 0.4436601255401917, \"estimate_description\": \"EM, blocked on: l.\\\"dob\\\" = r.\\\"dob\\\"\", \"estimated_probability_as_log_odds\": -0.32651161622835406, \"sql_condition\": \"ELSE\", \"comparison_level_label\": \"All other comparisons\", \"comparison_vector_value\": 0, \"comparison_name\": \"city\", \"comparison_sort_order\": 3}, {\"m_or_u\": \"m\", \"estimated_probability\": 0.5626002610028065, \"estimate_description\": \"EM, blocked on: (l.\\\"first_name\\\" = r.\\\"first_name\\\") AND (l.\\\"surname\\\" = r.\\\"surname\\\")\", \"estimated_probability_as_log_odds\": 0.36315786244208664, \"sql_condition\": \"\\\"email_l\\\" = \\\"email_r\\\"\", \"comparison_level_label\": \"Exact match on email\", \"comparison_vector_value\": 4, \"comparison_name\": \"email\", \"comparison_sort_order\": 4}, {\"m_or_u\": \"m\", \"estimated_probability\": 0.543332906314239, \"estimate_description\": \"EM, blocked on: l.\\\"dob\\\" = r.\\\"dob\\\"\", \"estimated_probability_as_log_odds\": 0.2506935883882106, \"sql_condition\": \"\\\"email_l\\\" = \\\"email_r\\\"\", \"comparison_level_label\": \"Exact match on email\", \"comparison_vector_value\": 4, \"comparison_name\": \"email\", \"comparison_sort_order\": 4}, {\"m_or_u\": \"m\", \"estimated_probability\": 0.21816795189628357, \"estimate_description\": \"EM, blocked on: (l.\\\"first_name\\\" = r.\\\"first_name\\\") AND (l.\\\"surname\\\" = r.\\\"surname\\\")\", \"estimated_probability_as_log_odds\": -1.8414195329946104, \"sql_condition\": \"NULLIF(regexp_extract(\\\"email_l\\\", '^[^@]+', 0), '') = NULLIF(regexp_extract(\\\"email_r\\\", '^[^@]+', 0), '')\", \"comparison_level_label\": \"Exact match on username\", \"comparison_vector_value\": 3, \"comparison_name\": \"email\", \"comparison_sort_order\": 4}, {\"m_or_u\": \"m\", \"estimated_probability\": 0.22358030063845905, \"estimate_description\": \"EM, blocked on: l.\\\"dob\\\" = r.\\\"dob\\\"\", \"estimated_probability_as_log_odds\": -1.7960436428022026, \"sql_condition\": \"NULLIF(regexp_extract(\\\"email_l\\\", '^[^@]+', 0), '') = NULLIF(regexp_extract(\\\"email_r\\\", '^[^@]+', 0), '')\", \"comparison_level_label\": \"Exact match on username\", \"comparison_vector_value\": 3, \"comparison_name\": \"email\", \"comparison_sort_order\": 4}, {\"m_or_u\": \"m\", \"estimated_probability\": 0.19518859679087333, \"estimate_description\": \"EM, blocked on: (l.\\\"first_name\\\" = r.\\\"first_name\\\") AND (l.\\\"surname\\\" = r.\\\"surname\\\")\", \"estimated_probability_as_log_odds\": -2.043781975575614, \"sql_condition\": \"jaro_winkler_similarity(\\\"email_l\\\", \\\"email_r\\\") >= 0.88\", \"comparison_level_label\": \"Jaro-Winkler distance of email >= 0.88\", \"comparison_vector_value\": 2, \"comparison_name\": \"email\", \"comparison_sort_order\": 4}, {\"m_or_u\": \"m\", \"estimated_probability\": 0.23307139250566425, \"estimate_description\": \"EM, blocked on: l.\\\"dob\\\" = r.\\\"dob\\\"\", \"estimated_probability_as_log_odds\": -1.718320348134645, \"sql_condition\": \"jaro_winkler_similarity(\\\"email_l\\\", \\\"email_r\\\") >= 0.88\", \"comparison_level_label\": \"Jaro-Winkler distance of email >= 0.88\", \"comparison_vector_value\": 2, \"comparison_name\": \"email\", \"comparison_sort_order\": 4}, {\"m_or_u\": \"m\", \"estimated_probability\": \"level not observed in training dataset\", \"estimate_description\": \"EM, blocked on: (l.\\\"first_name\\\" = r.\\\"first_name\\\") AND (l.\\\"surname\\\" = r.\\\"surname\\\")\", \"estimated_probability_as_log_odds\": null, \"sql_condition\": \"jaro_winkler_similarity(NULLIF(regexp_extract(\\\"email_l\\\", '^[^@]+', 0), ''), NULLIF(regexp_extract(\\\"email_r\\\", '^[^@]+', 0), '')) >= 0.88\", \"comparison_level_label\": \"Jaro-Winkler >0.88 on username\", \"comparison_vector_value\": 1, \"comparison_name\": \"email\", \"comparison_sort_order\": 4}, {\"m_or_u\": \"m\", \"estimated_probability\": \"level not observed in training dataset\", \"estimate_description\": \"EM, blocked on: l.\\\"dob\\\" = r.\\\"dob\\\"\", \"estimated_probability_as_log_odds\": null, \"sql_condition\": \"jaro_winkler_similarity(NULLIF(regexp_extract(\\\"email_l\\\", '^[^@]+', 0), ''), NULLIF(regexp_extract(\\\"email_r\\\", '^[^@]+', 0), '')) >= 0.88\", \"comparison_level_label\": \"Jaro-Winkler >0.88 on username\", \"comparison_vector_value\": 1, \"comparison_name\": \"email\", \"comparison_sort_order\": 4}, {\"m_or_u\": \"m\", \"estimated_probability\": 0.024043190310036607, \"estimate_description\": \"EM, blocked on: (l.\\\"first_name\\\" = r.\\\"first_name\\\") AND (l.\\\"surname\\\" = r.\\\"surname\\\")\", \"estimated_probability_as_log_odds\": -5.343117057489901, \"sql_condition\": \"ELSE\", \"comparison_level_label\": \"All other comparisons\", \"comparison_vector_value\": 0, \"comparison_name\": \"email\", \"comparison_sort_order\": 4}, {\"m_or_u\": \"m\", \"estimated_probability\": 1.540054163756538e-05, \"estimate_description\": \"EM, blocked on: l.\\\"dob\\\" = r.\\\"dob\\\"\", \"estimated_probability_as_log_odds\": -15.986637164535296, \"sql_condition\": \"ELSE\", \"comparison_level_label\": \"All other comparisons\", \"comparison_vector_value\": 0, \"comparison_name\": \"email\", \"comparison_sort_order\": 4}]}}, {\"mode\": \"vega-lite\"});\n",
              "</script>"
            ],
            "text/plain": [
              "alt.Chart(...)"
            ]
          },
          "execution_count": 12,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "linker.visualisations.parameter_estimate_comparisons_chart()"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "c44fcc26",
      "metadata": {},
      "source": [
        "### Saving the model\n",
        "\n",
        "We can save the model, including our estimated parameters, to a `.json` file, so we can use it in the next tutorial.\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 13,
      "id": "992703a7",
      "metadata": {
        "execution": {
          "iopub.execute_input": "2024-07-11T13:34:27.382784Z",
          "iopub.status.busy": "2024-07-11T13:34:27.382468Z",
          "iopub.status.idle": "2024-07-11T13:34:27.405586Z",
          "shell.execute_reply": "2024-07-11T13:34:27.404875Z"
        }
      },
      "outputs": [],
      "source": [
        "settings = linker.misc.save_model_to_json(\n",
        "    \"../demo_settings/saved_model_from_demo.json\", overwrite=True\n",
        ")"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "d07e6901-110f-449e-9534-8e24bbf1d5fb",
      "metadata": {},
      "source": [
        "## Detecting unlinkable records\n",
        "\n",
        "An interesting application of our trained model that is useful to explore before making any predictions is to detect 'unlinkable' records.\n",
        "\n",
        "Unlinkable records are those which do not contain enough information to be linked. A simple example would be a record containing only 'John Smith', and null in all other fields. This record may link to other records, but we'll never know because there's not enough information to disambiguate any potential links. Unlinkable records can be found by linking records to themselves - if, even when matched to themselves, they don't meet the match threshold score, we can be sure they will never link to anything.\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 14,
      "id": "b0f17f7c-fa83-41b5-b2da-25ae18e11d81",
      "metadata": {
        "execution": {
          "iopub.execute_input": "2024-07-11T13:34:27.409528Z",
          "iopub.status.busy": "2024-07-11T13:34:27.409234Z",
          "iopub.status.idle": "2024-07-11T13:34:28.142003Z",
          "shell.execute_reply": "2024-07-11T13:34:28.140889Z"
        }
      },
      "outputs": [
        {
          "data": {
            "text/html": [
              "\n",
              "<style>\n",
              "  #altair-viz-c6c9099114494a899c2a9acf2fef36a4.vega-embed {\n",
              "    width: 100%;\n",
              "    display: flex;\n",
              "  }\n",
              "\n",
              "  #altair-viz-c6c9099114494a899c2a9acf2fef36a4.vega-embed details,\n",
              "  #altair-viz-c6c9099114494a899c2a9acf2fef36a4.vega-embed details summary {\n",
              "    position: relative;\n",
              "  }\n",
              "</style>\n",
              "<div id=\"altair-viz-c6c9099114494a899c2a9acf2fef36a4\"></div>\n",
              "<script type=\"text/javascript\">\n",
              "  var VEGA_DEBUG = (typeof VEGA_DEBUG == \"undefined\") ? {} : VEGA_DEBUG;\n",
              "  (function(spec, embedOpt){\n",
              "    let outputDiv = document.currentScript.previousElementSibling;\n",
              "    if (outputDiv.id !== \"altair-viz-c6c9099114494a899c2a9acf2fef36a4\") {\n",
              "      outputDiv = document.getElementById(\"altair-viz-c6c9099114494a899c2a9acf2fef36a4\");\n",
              "    }\n",
              "    const paths = {\n",
              "      \"vega\": \"https://cdn.jsdelivr.net/npm/vega@5?noext\",\n",
              "      \"vega-lib\": \"https://cdn.jsdelivr.net/npm/vega-lib?noext\",\n",
              "      \"vega-lite\": \"https://cdn.jsdelivr.net/npm/vega-lite@5.17.0?noext\",\n",
              "      \"vega-embed\": \"https://cdn.jsdelivr.net/npm/vega-embed@6?noext\",\n",
              "    };\n",
              "\n",
              "    function maybeLoadScript(lib, version) {\n",
              "      var key = `${lib.replace(\"-\", \"\")}_version`;\n",
              "      return (VEGA_DEBUG[key] == version) ?\n",
              "        Promise.resolve(paths[lib]) :\n",
              "        new Promise(function(resolve, reject) {\n",
              "          var s = document.createElement('script');\n",
              "          document.getElementsByTagName(\"head\")[0].appendChild(s);\n",
              "          s.async = true;\n",
              "          s.onload = () => {\n",
              "            VEGA_DEBUG[key] = version;\n",
              "            return resolve(paths[lib]);\n",
              "          };\n",
              "          s.onerror = () => reject(`Error loading script: ${paths[lib]}`);\n",
              "          s.src = paths[lib];\n",
              "        });\n",
              "    }\n",
              "\n",
              "    function showError(err) {\n",
              "      outputDiv.innerHTML = `<div class=\"error\" style=\"color:red;\">${err}</div>`;\n",
              "      throw err;\n",
              "    }\n",
              "\n",
              "    function displayChart(vegaEmbed) {\n",
              "      vegaEmbed(outputDiv, spec, embedOpt)\n",
              "        .catch(err => showError(`Javascript Error: ${err.message}<br>This usually means there's a typo in your chart specification. See the javascript console for the full traceback.`));\n",
              "    }\n",
              "\n",
              "    if(typeof define === \"function\" && define.amd) {\n",
              "      requirejs.config({paths});\n",
              "      require([\"vega-embed\"], displayChart, err => showError(`Error loading script: ${err.message}`));\n",
              "    } else {\n",
              "      maybeLoadScript(\"vega\", \"5\")\n",
              "        .then(() => maybeLoadScript(\"vega-lite\", \"5.17.0\"))\n",
              "        .then(() => maybeLoadScript(\"vega-embed\", \"6\"))\n",
              "        .catch(showError)\n",
              "        .then(() => displayChart(vegaEmbed));\n",
              "    }\n",
              "  })({\"config\": {\"view\": {\"continuousWidth\": 400, \"continuousHeight\": 300}}, \"layer\": [{\"mark\": {\"type\": \"line\"}, \"encoding\": {\"x\": {\"axis\": {\"format\": \"+\", \"title\": \"Threshold match weight\"}, \"field\": \"match_weight\", \"type\": \"quantitative\"}, \"y\": {\"axis\": {\"format\": \"%\", \"title\": \"Percentage of unlinkable records\"}, \"field\": \"cum_prop\", \"type\": \"quantitative\"}}}, {\"mark\": {\"type\": \"point\"}, \"encoding\": {\"opacity\": {\"condition\": {\"param\": \"x_match_weight_y_cum_prop_coords_of_mouse\", \"value\": 1, \"empty\": false}, \"value\": 0}, \"tooltip\": [{\"field\": \"match_weight\", \"format\": \"+.5\", \"title\": \"Match weight\", \"type\": \"quantitative\"}, {\"field\": \"match_probability\", \"format\": \".5\", \"title\": \"Match probability\", \"type\": \"quantitative\"}, {\"field\": \"cum_prop\", \"format\": \".3%\", \"title\": \"Proportion of unlinkable records\", \"type\": \"quantitative\"}], \"x\": {\"axis\": {\"title\": \"Threshold match weight\"}, \"field\": \"match_weight\", \"type\": \"quantitative\"}, \"y\": {\"axis\": {\"format\": \"%\", \"title\": \"Percentage of unlinkable records\"}, \"field\": \"cum_prop\", \"type\": \"quantitative\"}}, \"name\": \"mouse_coords\"}, {\"mark\": {\"type\": \"rule\", \"color\": \"gray\"}, \"encoding\": {\"x\": {\"field\": \"match_weight\", \"type\": \"quantitative\"}}, \"transform\": [{\"filter\": {\"param\": \"x_match_weight_y_cum_prop_coords_of_mouse\", \"empty\": false}}]}, {\"mark\": {\"type\": \"rule\", \"color\": \"gray\"}, \"encoding\": {\"y\": {\"field\": \"cum_prop\", \"type\": \"quantitative\"}}, \"transform\": [{\"filter\": {\"param\": \"x_match_weight_y_cum_prop_coords_of_mouse\", \"empty\": false}}]}], \"data\": {\"name\": \"data-6a3e3ca5c2af2f39ba3584a28942c6e4\"}, \"height\": 400, \"params\": [{\"name\": \"x_match_weight_y_cum_prop_coords_of_mouse\", \"select\": {\"type\": \"point\", \"fields\": [\"match_weight\", \"cum_prop\"], \"nearest\": true, \"on\": \"mouseover\"}, \"views\": [\"mouse_coords\"]}], \"title\": {\"text\": \"Unlinkable records\", \"subtitle\": \"Records with insufficient information to exceed a given match threshold\"}, \"width\": 400, \"$schema\": \"https://vega.github.io/schema/vega-lite/v5.9.3.json\", \"datasets\": {\"data-6a3e3ca5c2af2f39ba3584a28942c6e4\": [{\"match_weight\": -0.58, \"match_probability\": 0.40099, \"prop\": 0.0020000000949949026, \"cum_prop\": 0.0020000000949949026}, {\"match_weight\": 3.29, \"match_probability\": 0.90709, \"prop\": 0.0010000000474974513, \"cum_prop\": 0.003000000142492354}, {\"match_weight\": 4.09, \"match_probability\": 0.9445, \"prop\": 0.0010000000474974513, \"cum_prop\": 0.004000000189989805}, {\"match_weight\": 4.45, \"match_probability\": 0.95627, \"prop\": 0.0010000000474974513, \"cum_prop\": 0.0050000002374872565}, {\"match_weight\": 4.51, \"match_probability\": 0.95795, \"prop\": 0.0010000000474974513, \"cum_prop\": 0.006000000284984708}, {\"match_weight\": 5.09, \"match_probability\": 0.97146, \"prop\": 0.0010000000474974513, \"cum_prop\": 0.007000000332482159}, {\"match_weight\": 5.91, \"match_probability\": 0.98367, \"prop\": 0.0010000000474974513, \"cum_prop\": 0.00800000037997961}, {\"match_weight\": 5.94, \"match_probability\": 0.98393, \"prop\": 0.0010000000474974513, \"cum_prop\": 0.009000000427477062}, {\"match_weight\": 6.19, \"match_probability\": 0.98649, \"prop\": 0.0010000000474974513, \"cum_prop\": 0.010000000474974513}, {\"match_weight\": 6.31, \"match_probability\": 0.98757, \"prop\": 0.0020000000949949026, \"cum_prop\": 0.012000000569969416}, {\"match_weight\": 6.61, \"match_probability\": 0.98983, \"prop\": 0.0010000000474974513, \"cum_prop\": 0.013000000617466867}, {\"match_weight\": 7.01, \"match_probability\": 0.99229, \"prop\": 0.0010000000474974513, \"cum_prop\": 0.014000000664964318}, {\"match_weight\": 7.09, \"match_probability\": 0.99274, \"prop\": 0.0010000000474974513, \"cum_prop\": 0.01500000071246177}, {\"match_weight\": 7.19, \"match_probability\": 0.9932, \"prop\": 0.0010000000474974513, \"cum_prop\": 0.01600000075995922}, {\"match_weight\": 7.27, \"match_probability\": 0.99357, \"prop\": 0.0010000000474974513, \"cum_prop\": 0.017000000807456672}, {\"match_weight\": 7.3, \"match_probability\": 0.99369, \"prop\": 0.0010000000474974513, \"cum_prop\": 0.018000000854954123}, {\"match_weight\": 7.5, \"match_probability\": 0.9945, \"prop\": 0.0010000000474974513, \"cum_prop\": 0.019000000902451575}, {\"match_weight\": 7.59, \"match_probability\": 0.99485, \"prop\": 0.003000000026077032, \"cum_prop\": 0.022000000928528607}, {\"match_weight\": 7.9, \"match_probability\": 0.99582, \"prop\": 0.003000000026077032, \"cum_prop\": 0.02500000095460564}, {\"match_weight\": 7.91, \"match_probability\": 0.99587, \"prop\": 0.0010000000474974513, \"cum_prop\": 0.02600000100210309}, {\"match_weight\": 8.19, \"match_probability\": 0.99659, \"prop\": 0.003000000026077032, \"cum_prop\": 0.029000001028180122}, {\"match_weight\": 8.26, \"match_probability\": 0.99674, \"prop\": 0.003000000026077032, \"cum_prop\": 0.032000001054257154}, {\"match_weight\": 8.41, \"match_probability\": 0.99707, \"prop\": 0.0010000000474974513, \"cum_prop\": 0.033000001101754606}, {\"match_weight\": 8.59, \"match_probability\": 0.99742, \"prop\": 0.0010000000474974513, \"cum_prop\": 0.03400000114925206}, {\"match_weight\": 8.76, \"match_probability\": 0.99769, \"prop\": 0.0010000000474974513, \"cum_prop\": 0.03500000119674951}, {\"match_weight\": 8.83, \"match_probability\": 0.9978, \"prop\": 0.0010000000474974513, \"cum_prop\": 0.03600000124424696}, {\"match_weight\": 9.3, \"match_probability\": 0.99841, \"prop\": 0.0010000000474974513, \"cum_prop\": 0.03700000129174441}, {\"match_weight\": 9.44, \"match_probability\": 0.99856, \"prop\": 0.0010000000474974513, \"cum_prop\": 0.03800000133924186}, {\"match_weight\": 9.48, \"match_probability\": 0.9986, \"prop\": 0.0010000000474974513, \"cum_prop\": 0.039000001386739314}, {\"match_weight\": 9.5, \"match_probability\": 0.99862, \"prop\": 0.0020000000949949026, \"cum_prop\": 0.041000001481734216}, {\"match_weight\": 9.59, \"match_probability\": 0.99871, \"prop\": 0.0020000000949949026, \"cum_prop\": 0.04300000157672912}, {\"match_weight\": 9.73, \"match_probability\": 0.99883, \"prop\": 0.0010000000474974513, \"cum_prop\": 0.04400000162422657}, {\"match_weight\": 9.83, \"match_probability\": 0.9989, \"prop\": 0.0010000000474974513, \"cum_prop\": 0.04500000167172402}, {\"match_weight\": 9.89, \"match_probability\": 0.99895, \"prop\": 0.0010000000474974513, \"cum_prop\": 0.04600000171922147}, {\"match_weight\": 9.94, \"match_probability\": 0.99898, \"prop\": 0.0010000000474974513, \"cum_prop\": 0.047000001766718924}, {\"match_weight\": 10.12, \"match_probability\": 0.9991, \"prop\": 0.0010000000474974513, \"cum_prop\": 0.048000001814216375}, {\"match_weight\": 10.18, \"match_probability\": 0.99914, \"prop\": 0.0010000000474974513, \"cum_prop\": 0.04900000186171383}, {\"match_weight\": 10.27, \"match_probability\": 0.99919, \"prop\": 0.0010000000474974513, \"cum_prop\": 0.05000000190921128}, {\"match_weight\": 10.36, \"match_probability\": 0.99924, \"prop\": 0.0020000000949949026, \"cum_prop\": 0.05200000200420618}, {\"match_weight\": 10.45, \"match_probability\": 0.99928, \"prop\": 0.0010000000474974513, \"cum_prop\": 0.05300000205170363}, {\"match_weight\": 10.53, \"match_probability\": 0.99932, \"prop\": 0.0010000000474974513, \"cum_prop\": 0.05400000209920108}, {\"match_weight\": 10.56, \"match_probability\": 0.99934, \"prop\": 0.0010000000474974513, \"cum_prop\": 0.055000002146698534}, {\"match_weight\": 10.59, \"match_probability\": 0.99935, \"prop\": 0.0010000000474974513, \"cum_prop\": 0.056000002194195986}, {\"match_weight\": 10.61, \"match_probability\": 0.99936, \"prop\": 0.0010000000474974513, \"cum_prop\": 0.05700000224169344}, {\"match_weight\": 10.64, \"match_probability\": 0.99937, \"prop\": 0.0010000000474974513, \"cum_prop\": 0.05800000228919089}, {\"match_weight\": 10.76, \"match_probability\": 0.99942, \"prop\": 0.0010000000474974513, \"cum_prop\": 0.05900000233668834}, {\"match_weight\": 10.94, \"match_probability\": 0.99949, \"prop\": 0.0010000000474974513, \"cum_prop\": 0.06000000238418579}, {\"match_weight\": 10.95, \"match_probability\": 0.9995, \"prop\": 0.0010000000474974513, \"cum_prop\": 0.06100000243168324}, {\"match_weight\": 11.06, \"match_probability\": 0.99953, \"prop\": 0.0010000000474974513, \"cum_prop\": 0.062000002479180694}, {\"match_weight\": 11.1, \"match_probability\": 0.99955, \"prop\": 0.0010000000474974513, \"cum_prop\": 0.06300000252667814}, {\"match_weight\": 11.15, \"match_probability\": 0.99956, \"prop\": 0.0010000000474974513, \"cum_prop\": 0.0640000025741756}, {\"match_weight\": 11.27, \"match_probability\": 0.9996, \"prop\": 0.0010000000474974513, \"cum_prop\": 0.06500000262167305}, {\"match_weight\": 11.34, \"match_probability\": 0.99961, \"prop\": 0.0020000000949949026, \"cum_prop\": 0.06700000271666795}, {\"match_weight\": 11.54, \"match_probability\": 0.99966, \"prop\": 0.0010000000474974513, \"cum_prop\": 0.0680000027641654}, {\"match_weight\": 11.76, \"match_probability\": 0.99971, \"prop\": 0.0010000000474974513, \"cum_prop\": 0.06900000281166285}, {\"match_weight\": 11.8, \"match_probability\": 0.99972, \"prop\": 0.0010000000474974513, \"cum_prop\": 0.0700000028591603}, {\"match_weight\": 11.98, \"match_probability\": 0.99975, \"prop\": 0.003000000026077032, \"cum_prop\": 0.07300000288523734}, {\"match_weight\": 12.25, \"match_probability\": 0.99979, \"prop\": 0.003000000026077032, \"cum_prop\": 0.07600000291131437}, {\"match_weight\": 12.37, \"match_probability\": 0.99981, \"prop\": 0.0010000000474974513, \"cum_prop\": 0.07700000295881182}, {\"match_weight\": 12.47, \"match_probability\": 0.99982, \"prop\": 0.004999999888241291, \"cum_prop\": 0.08200000284705311}, {\"match_weight\": 12.56, \"match_probability\": 0.99983, \"prop\": 0.0010000000474974513, \"cum_prop\": 0.08300000289455056}, {\"match_weight\": 12.73, \"match_probability\": 0.99985, \"prop\": 0.0020000000949949026, \"cum_prop\": 0.08500000298954546}, {\"match_weight\": 12.84, \"match_probability\": 0.99986, \"prop\": 0.004000000189989805, \"cum_prop\": 0.08900000317953527}, {\"match_weight\": 12.93, \"match_probability\": 0.99987, \"prop\": 0.003000000026077032, \"cum_prop\": 0.0920000032056123}, {\"match_weight\": 13.08, \"match_probability\": 0.99988, \"prop\": 0.006000000052154064, \"cum_prop\": 0.09800000325776637}, {\"match_weight\": 13.15, \"match_probability\": 0.99989, \"prop\": 0.004000000189989805, \"cum_prop\": 0.10200000344775617}, {\"match_weight\": 13.35, \"match_probability\": 0.9999, \"prop\": 0.004000000189989805, \"cum_prop\": 0.10600000363774598}, {\"match_weight\": 13.48, \"match_probability\": 0.99991, \"prop\": 0.004000000189989805, \"cum_prop\": 0.11000000382773578}, {\"match_weight\": 13.68, \"match_probability\": 0.99992, \"prop\": 0.009999999776482582, \"cum_prop\": 0.12000000360421836}, {\"match_weight\": 13.86, \"match_probability\": 0.99993, \"prop\": 0.004999999888241291, \"cum_prop\": 0.12500000349245965}, {\"match_weight\": 14.15, \"match_probability\": 0.99994, \"prop\": 0.004000000189989805, \"cum_prop\": 0.12900000368244946}, {\"match_weight\": 14.41, \"match_probability\": 0.99995, \"prop\": 0.013000000268220901, \"cum_prop\": 0.14200000395067036}, {\"match_weight\": 14.79, \"match_probability\": 0.99996, \"prop\": 0.013000000268220901, \"cum_prop\": 0.15500000421889126}, {\"match_weight\": 15.28, \"match_probability\": 0.99997, \"prop\": 0.02199999988079071, \"cum_prop\": 0.17700000409968197}, {\"match_weight\": 16.02, \"match_probability\": 0.99998, \"prop\": 0.03400000184774399, \"cum_prop\": 0.21100000594742596}, {\"match_weight\": 17.57, \"match_probability\": 0.99999, \"prop\": 0.07599999755620956, \"cum_prop\": 0.2870000035036355}]}}, {\"mode\": \"vega-lite\"});\n",
              "</script>"
            ],
            "text/plain": [
              "alt.LayerChart(...)"
            ]
          },
          "execution_count": 14,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "linker.evaluation.unlinkables_chart()"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "ca2570b7-9f68-4fb2-b7a9-527233a8fcd4",
      "metadata": {},
      "source": [
        "In the above chart, we can see that about 1.3% of records in the input dataset are unlinkable at a threshold match weight of 6.11 (correponding to a match probability of around 98.6%)\n"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "83fd8e7f",
      "metadata": {},
      "source": [
        "!!! note \"Further Reading\"\n",
        "\n",
        "    :material-tools: For more on the model estimation tools in Splink, please refer to the [Model Training API documentation](../../api_docs/training.md).\n",
        "\n",
        "    :simple-readme: For a deeper dive on:\n",
        "\n",
        "    * choosing comparisons, please refer to the [Comparisons Topic Guides](../../topic_guides/comparisons/customising_comparisons.ipynb)\n",
        "    * the underlying model theory, please refer to the [Fellegi Sunter Topic Guide](../../topic_guides/theory/fellegi_sunter.md)\n",
        "    * model training, please refer to the Model Training Topic Guides (Coming Soon).\n",
        "\n",
        "    :bar_chart: For more on the charts used in this tutorial, please refer to the [Charts Gallery](../../charts/index.md).\n"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "fd531cd2",
      "metadata": {},
      "source": [
        "## Next steps\n",
        "\n",
        "Now we have trained a model, we can move on to using it predict matching records.\n"
      ]
    }
  ],
  "metadata": {
    "kernelspec": {
      "display_name": "Python 3 (ipykernel)",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.10.8"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 5
}
